Fuzzy multiple criteria selection of
government-sponsored frontier
technology R&D projects
Yeou-Geng Hsu, Gwo-Hshing Tzeng and
Joseph Z. Shyu
Institute of Technology Management, National Chiao Tung University, 1001, Ta-Hsueh Rd., Hsinchu, Taiwan. yghsu@hermes.com.tw
Selection of government-sponsored frontier R&D projects is made difficult by the coexistence of the conflicting participating parties, the availability of experts for new frontier technology review, and the ambiguity of new frontier technology. This paper presents a model that includes (1) using the analytical hierarchy process (AHP) method to integrate various expectations from different interest groups into evaluating objectives/criteria, (2) the group-decision method by technical experts based on the predetermined objectives/criteria, and (3) the fuzzy approach in scoring the subjective judgments of the experts. The results reveal that differences of weights toward each criterion exist among various groups. The government and academia care more about social benefits, the researchers are more concerned about intellectual properties, and the experts from industry emphasize the importance of feasibility. The method presented in this paper was applied at a national research institute in Taiwan. The results reveal that: (1) the approach can solve the disparity between the profound knowledge required for evaluation and the different expectation from various interest groups, (2) the fuzzy approach is suitable to frontier technology R&D project selection because of the vagueness of the nature of frontier technology and the difficulties in evaluating quantitatively and accurately.
1. Introduction
M
any multiple criteria models have been used for R&D project selection. However, there has been little research on the selection of government-sponsored technology R&D pro-jects. For developing countries, it is especially critical to focus on technology development due to their limited resources. More than 97% of the companies in Taiwan are small to medium sized firms. They do not have extensive R&D capabil-ities. Taiwan has thus adopted the use of non-profit applied research institutes to develop generic technology, and subsequently transfer the results of such research to the industrialsectors (Chang and Hsu, 1997; Mathews, 2002). Using this approach, success has been demon-strated in high-tech sector development, such as the IC industry and opto-electronics industry. During the 1990s, the R&D funding from the Ministry of Economic Affairs (MOEA) ac-counted for nearly 10% of Taiwan’s total R&D expenditures. In 2001, for example, the MOEA sponsored some US$420 million in technology R&D projects. To ensure that the technologies developed by the research institutes are commer-cially applicable in the industrial sector, the MOEA invites experts from industry, govern-ment and academia to review the feasibility and the expected returns of these research proposals.
In general, the members of review committees are chosen from different interest groups and the selection processes are usually conducted by group decision-making with no clearly defined criteria for project selection. Therefore, the review committee tends to select projects in a consensus way with lots of compromise.
In the past decade, as Taiwanese high-tech firms have been becoming more competitive, the Taiwanese government has gradually adjusted its science and technology policy from a product-orientation to a more technological focus. The weight of the ‘New Frontier Technology R&D Projects (NFTPs)’ has increased dramatically since 1998, and the importance of innovation has been constantly strengthened. The ‘New Frontier Technology’ represents the technology still at an early stage of its development, for which there are not yet any commercial products. However, the goal of a NFTP is not to explore knowledge. It is a mission-oriented project with clearly stated objectives aiming ultimately at economic success. Three characteristics of this kind of research project make the selection even more difficult. First of all, the nature of govern-ment-supported projects, for which various ex-pectations from different interest groups must be taken into account. The interested parties include government, research organizations, and indus-try. Second, a relatively weak knowledge base in frontier technology areas exists in developing countries like Taiwan. Third, the vagueness of new frontier technology, its technical uncertain-ties, market risks, lack of hard data, and lack of qualified evaluators are part of the reasons why evaluation usually proceeds subjectively and intuitively. Therefore, the major purpose of this paper is to solve the disparity between the profound knowledge required for evaluation and the different expectations from various interest groups.
Considering the availability of local experts of frontier technology, the MOEA has delegated the review process to selected research institutes since 1999. That is, research institutes proposed a project and reviewed it by themselves. For instance, an advisory committee was organized within the Industrial Technology Research In-stitute (ITRI) to review the proposals from laboratories. The members of the advisory committee are chosen from world-class experts, and most of them are overseas Chinese. The committee members are no doubt more capable in terms of domain knowledge than local reviewers. But for a government-sponsored
pro-ject, it might be more important to have a result acceptable to different interest groups rather than a good technical outcome. In order to solve this dilemma, we propose a fuzzy multiple criteria approach. The approach includes a group deci-sion-making method to incorporate different opinions into a single objective, an analytical hierarchy process (AHP) method (Lockett et al., 1986; Saaty, 1980) to obtain the evaluating criteria and their weights, a triangular fuzzy number (TFN) for scoring an expert’s judgment, and the best non-fuzzy performance number (BNP) method to synthesize the group decision and to rank the projects (Teng and Tzeng, 1996; Tsaur et al., 1997). Finally two illustrative examples are presented to demonstrate this approach.
This paper is organized as follows: In section 1, the problems and purposes of the research are described. In section 2, the background for R&D project selection is introduced. In section 3, the research methodology of the fuzzy multi-criteria selection model is proposed. In section 4, a hierarchy model for R&D project selection and the evaluating criteria formation are presented and two examples from ITRI are demonstrated, and the results discussed. Finally conclusions are presented in section 5.
2. Background on R&D project selection
Hundreds of methods and techniques exist in the literature for R&D project selection. Approaches tend to be either qualitative or quantitative, ranging from unstructured peer review to sophis-ticated mathematical programming. Overviews on the topic of R&D project selection are founded in Baker (1974), Baker and Freeland (1975), Baker and Pound (1964), Danila (1989), Gaynor (1990), Henriksen and Traynor (1999), Liberatore and Titus (1983), Martino (1995), Schmidt and Freeland (1992), and Steele (1988).
R&D project selection methods can be usually placed into one of the following categories (Henriksen and Traynor, 1999): (1) mathematical programming and portfolio optimization, includ-ing integer programminclud-ing, linear programminclud-ing, nonlinear programming, goal programming, dy-namic programming, and portfolio optimization. (2) Economic models, such as net present value, internal return rate, return on investment, cost-benefit analysis, and option pricing theory. (3) Decision analysis, including multi-attribute utility theory, decision tree, risk analysis, AHP, and
scoring. (4) Interactive methods, such as Delphi, Q-sort, behaviour decision aids, and decentra-lized hierarchical modeling. (5) Artificial intelli-gence, including expert systems, and fuzzy set approach. However, few methods have gained wide acceptance in the real world (Liberatore and Titus, 1983; Schmidt and Freeland, 1992). The recent trend has been to combine the different approaches into an integrated way that is appropriate for a particular situation (Fahrni and Spa¨tig, 1990).
Most of the research on R&D project selection concentrated on private sectors whilst little research has been done on government-sponsored R&D projects. Methodologies similar to those used by private firms, such as cost-benefit analysis and option pricing, have also been used for government-sponsored activities (Bergman and Mark, 2002; Henriksen and Traynor, 1999; Vonortas and Hertzfeld, 1998). However, a government-sponsored project differs from that of the private sector in two major aspects: (1) government-sponsored R&D is by nature a strategic and long-term investment. It is esstially about demonstrating opportunities to en-able the private sector to make a better investment in a potentially profitable technologi-cal field. Conventional financial justification approaches are probably inadequate; (2) political factors and interest parties always influence the allocation of R&D resources in the public sector. The selection of government-sponsored projects has to be an ‘open and fair’ process. The difficulty in selecting the new frontier technology R&D projects in Taiwan is further increased by the coexistence of the lack of experts for frontier technology review, and the ambiguity of frontier technology itself.
Meade and Presley (2002) indicated that there are three major themes relating to R&D project selection: (1) the need to relate selection criteria to organizational strategy; (2) the need to consider qualitative benefits and risks of candi-date projects; (3) the need to reconcile and integrate the needs and desires of different stakeholders. The AHP method for decision analysis was first introduced by Saaty and has been used for project selection and prioritization (Alidi, 1996; Brenner, 1994; Islei et al., 1991; Liberatore, 1987; Lockett et al., 1986). The AHP method is an efficient measurement and a multi-objective decision-making approach that employs pair comparison to determine the weights and priorities of a variety of factors, attributes, elements and alternatives. Brussion (1980)
indi-cated that one of the key functions of the project-selection process is to build commitment and consensus. The advantages of AHP are recog-nized as its ease of use, intuitiveness, and consensus building (Alidi, 1996).
For new frontier technology R&D pro-grammes, where the objectives are radically advanced technologies and ultimately economic impacts, the criteria are difficult to quantify and the outcome is highly uncertain. R&D project selection models that permit intuitive judgment tend to be more acceptable from a practitioner-oriented point of view. Since Zadeh (1965) first introduced fuzzy set theory, and subsequently the fuzzy decision-making method (Bellman and Zadeh, 1970) in fuzzy environments, many other studies have dealt with uncertain fuzzy problems by applying fuzzy set theory (Chen and Hwang, 1992; Coffin and Taylor, 1996; Dias 1988; Teng and Tzeng, 1996). The fuzzy approach is thus suitable to scoring a project that is based on subjective judgments of evaluators.
3. A fuzzy multiple criteria approach for
R&D project selection
A hierarchy model for R&D project selection is first proposed and further revised according to the opinion of 20 experts from research organiza-tions, industry and academia, and the criteria weights are obtained by using AHP. Then a separate group of technical experts review the R&D projects according to the determined criteria, and the fuzzy approach is used to score the subjective judgments of the experts for the performance value of each alternative project. Finally we rank the projects by using the best non-fuzzy performance number (BNP) to synthe-size the group decision.
3.1. Evaluating the weight for the
hierarchy system
The evaluators determined the AHP weightings by conducting the pair-wise comparisons between two criteria/objectives. Saaty (1980) used the principal eigenvector of the pair-wise comparison matrix derived from the scaling ratios to find the relative weight importance among the criteria/ objectives of the hierarchy system.
Suppose there is a set of n criteria/objectives in pairs according to their relative weight (impor-tance) scaling. Denote the criteria/objectives by c1, c2, y ,cnand their weights by w1, w2, y ,wn.If
w¼ (w1, w2, y ,wn)t is given, a matrix A of the following equation can represent the pair-wise comparisons,
ðA lmaxIÞw ¼ 0
Where A is the matrix of the pair-wise comparison value derived from the intuitive judgments. Then we can find the eigenvector w with its lmax which satisfies Aw¼ lmaxw. It has been shown that humans usually judge with a certain degree of inconsistency. Saaty (1980) used the consistency index (C.I.) as an indicator of ‘closeness to consistency’,
C:I :¼ ðlmax nÞ=ðn 1Þ
The derived value is zero if the evaluators’ judgment is totally consistent and is one if the judgments are completely inconsistent. In gener-al, the value of lmaxcan be accepted if C.I. is not greater than 0.1. Several software packages are available to assist in conducting pair-wise com-parison such as Expert Choice.
3.2. The performance value of the
alternatives
‘Extremely important’, ‘not very cold’, ‘probable so’; these terms of expression can be heard very often in daily life, and their commonality is that they are more or less tainted with uncertainty. With such an idea in mind, this study considers the possible fuzzy subjective judgment of the evaluators during the project evaluation. The applications of fuzzy theory can be described as follows:
a. Fuzzy numbers. According to the definition of Dubis and Prade (1978; 1980), the fuzzy number ~AA is a fuzzy set, and its membership function is mAA~ðxÞ :R ! ½0; 1 where x represents the R&D projects. It is common to use triangular fuzzy numbers (TFNs) mAA~ðxÞ ¼ ðL; M; UÞ for fuzzy operations, as shown in equation (1) and Figure 1 mAA~ðxÞ ¼ ðx LÞ=ðM LÞ; L x M ðU xÞ=ðU MÞ; M x U 0; otherwise 2 4 ð1Þ
b. Linguistic variable. A linguistic variable is a variable whose values are words or sentences in natural or artificial language. For example, the expressions of objectives/criteria such as ‘techno-logical competitiveness’ and ‘benefits for human
life’ represent linguistic variables in the context of these problems. The linguistic variable may take on such values as ‘very high’, ‘high’, ‘fair’, ‘low’ or ‘very low’. We utilized a set of triangular fuzzy numbers within the scale range of 0–100 to represent the value. The evaluators first assigned their subjective scores to the linguistic variables and then conducted their judgments. For in-stance, an evaluator assigned a set of TFNs (80, 90, 100) for ‘very high’, (55, 70, 85) for ‘high’, (35, 50, 65) for ‘fair’, and so on. When the evaluator rated project B as ‘fair’ toward criterion ‘techno-logical competitiveness’, then a set of TFNs (35, 50, 65) was thus representing his/her ‘fair’ in the mathematical processing.
3.3. Evaluating the R&D projects
The method and procedure of evaluation are summarized as follows:
a. Measuring objectives/criteria. Let ~EEk ij be the fuzzy performance value of kth evaluator toward project i under objective/criterion j, and let the performance of the objectives/criteria be indi-cated by a set S,
~ E
Eijk¼ ðLEkij; MEijk; UEijkÞ; j 2 S
Since the perception of each evaluator varies according to their knowledge and experience, the definitions of the linguistic variables also vary. Thus, this study uses the notion of average value so as to integrate the fuzzy judgment values of m evaluators, that is,
~ E
Eij¼ ð1=mÞ ð ~EE1ij ~EE2ij . . . ~EEijmÞ Where the sign# denotes fuzzy multiplication; the sign " denotes fuzzy addition; ~EEij is the average fuzzy number of the judgment of m evaluators, and a triangular fuzzy number can display it as follows:
~ E
Eij¼ ðLEij; MEij; UEijÞ
L M U x
Membership Function
Figure 1. Membership function of the triangular fuzzy num-ber.
The proceeding end-point values LEij¼ ð1=mÞðPmk¼1LEk ijÞ; MEij¼ ð1=mÞð Pm k¼1MEijkÞ and UEij¼ ð1=mÞðPmk¼1UEk ijÞ
b. Fuzzy synthetic decision. The weights of the objectives/criteria and the fuzzy performance values must be integrated by the operation of fuzzy numbers. According to the weights wj derived by AHP method, we get a weight vector w, and then the fuzzy performance matrix ~EE of each of the projects can be obtained from the fuzzy performance value of each project under n objectives/criteria, that is,
w¼ ðw1; w2; . . . ; wnÞt ~ E E¼ ð ~EEijÞ ~ R R¼ ~EE *w
The sign ‘*’ indicates the operation of the fuzzy numbers, including fuzzy addition and plication. Because the operation of fuzzy multi-plication is relatively complex, it is usually denoted by an approximate multiplied result ~RR that is a fuzzy number ( ~RR¼ ~RR1; . . . ; ~RRi; . . . ; ~RRI). It can be expressed as follows:
Ri¼ ðLRi; MRi; URiÞ; 8i LRi¼X n j¼1 LEijwj MRi¼X n j¼1 MEijwj URi¼ Xn j¼1 UEijwj
c. Ranking the projects. The result of the fuzzy synthetic decision reached by each project is a fuzzy number. Therefore, it is necessary to transform a fuzzy number into a non-fuzzy number in order to rank the projects. In many research projects the procedure for de-fuzzifica-tion is to locate the Best Non-fuzzy Performance (BNP) value. Methods of such de-fuzzified fuzzy ranking include mean of maximal (MON), centre of area (COA), and a-Cut (Teng and Tzeng, 1996; Zhau and Goving, 1991). To utilize the COA method to determine the BNP is simple and practical. The BNP value of the fuzzy number can be calculated as follows:
BNPi¼ LRi
þ ðURi½ LRiÞ þ ðMRi LRiÞ=3; 8i
The projects can then be ranked according to their BNP value.
4. Evaluating the hierarchy system for
R&D project selection
As ITRI was the first institute empowered to conduct ‘Frontier Technology R&D Projects’ in Taiwan, we first proposed over 30 criteria for R&D project selection based on a literature review and a survey of ITRI’s existing approach.
4.1. Evaluating hierarchy model and its
criteria
We then invited 20 experts including senior project managers and technical leaders from ITRI and industry to review the hierarchy. The hierarchy project selection model was then con-structed as Figure 2, with three aspects of goal; benefits, technology and execution.
(1) Benefits: The benefits that may accrue to the whole nation after the research results are realized, including economic and social ben-efits.
(2) Technology: The impacts of targeted technol-ogy developed in the research project, includ-ing the technological competitiveness, and the relevance of technology.
(3) Execution: The execution of the project and the implementation of the research results, including the feasibility of project execution, and the success rate of commercialization. The evaluation criteria are summarized in Table 1.
4.2. Weights of evaluation criteria/
objectives and their implications
Three groups of evaluators, from the government agency (GA), the industry (IN), and the research institute (RI), were invited to conduct the AHP weighting process. They were requested to respond to a questionnaire by comparing the relative importance of criteria pair-wise. We used a scale range of 1–9 to represent relative importance. Where there was inconsistency, the evaluators were asked to repeat their comparison process until the consistency index was less than 0.1. Weights of each group toward the evaluating objectives and criteria are summarized in Tables 2 and 3.In general, the weights of objectives fall in the range from 0.16 to 0.20 with the exception of the weight of ‘social benefits’ (0.082). It is noteworthy that (1) both ‘technology aspect’ (0.372) and ‘execution aspect’ (0.383) are more important than ‘benefit aspect’ (0.245), which can be explained by the nature of frontier technology projects. The perceived uncertainties and risks of a frontier technology project will cause the rise of the weights of importance of ‘Execution’ and ‘Technology’, whereas the opposite trend is observed as a result of a lower weight of benefit. (2) Economical benefit (0.164) is more important than social benefit (0.082), reflecting the fact that MOEA put more emphasis on economic benefits than on social benefits.
Among the criteria, the average weight of ‘innovativeness of research idea’ (0.084) is the highest, followed by ‘extendibility’ (0.068), ‘cap-ability of research team’ (0.066), ‘advancement of technology’ (0.060), ‘proprietary technology’ (0.059), ‘technology connection’ (0.059), and ‘timing’ (0.058). It is rational that a unique and innovative idea is much more important than the
proper choice of research topics in highly competitive technology market.
Furthermore, the weights toward each criterion are different among various groups. In general, the opinions are more diversified in ‘benefit for human life’, ‘market scope of application’ and ‘advancement of technology’, whereas there is higher consensus for ‘innovativeness of idea’ and ‘value added of target products’. It can be noted that:
(1) For experts from government and academia, the highest criteria scores are ‘innovativeness of idea’ (0.078), ‘benefits to human life’ (0.064), and ‘extendibility’ (0.062), followed by ‘timing’ (0.059) and ‘favourable environ-ment’ (0.057). The weight of ‘benefits to human life’ (0.064) towards GA group is higher than the weights toward the other two groups.
(2) For experts from industries, the highest criteria scores are ‘innovativeness of idea’ (0.095), ‘capabilities of research team’ (0.073) and ‘extendibility’ (0.070), followed by ‘safety
Goal Aspects Objectives Criteria Market scope of application Growth potential of product Value-added of target products Relatedness of industry Improvements on QESIS Concatenation with S&T policy Benefits for human life
Innovativeness Advancement of technology Proprietary technology Generics or specific Technological connections Extendibility
Soundness of scientific principles Quality of proposal
Capability of research team Safety and pollution concerns Intensity of competition Favorable environments Availability of complementary Assets
Timing
Selection for Frontier Technology R&D Project
Economic benefits Social benefits Competitiveness Relevance Feasibility Success rate Technology Execution Benefits
and pollution concerns’ (0.067), ‘timing’ (0.067), ‘generics or specific’ (0.059) and ‘intensity of competition’ (0.057). It is
reason-able that the industrial sectors are more concerned about safety and environmental issues than the other two groups.
Table 1. Criteria for Evaluation.
Criteria Description
Market scope of applications The potential market size for the targeted products Growth of targeted products The growth potential of the targeted product applications Value-added of the targeted
products
The value-added potential for the targeted products
Industry relatedness The scope of industry to which the technology developed can be applied Improvements on QESIS Benefits to society through the improvement in quality, environmental protection,
industrial safety, national image and industrial standards Concatenation with S&T
policy
The concatenation of the project with the science and technology policy of the nation
Benefits for human life The benefits for human life, such as health, and quality of life
Innovativeness of idea How innovative is the research idea? Is it an incremental improvement or a radical innovation?
Advancement of technology How advanced is the targeted technology compared with existing technology Proprietary technology Will the project generate a proprietary technology position through the
intellectual property rights?
Generics or specific Is the technology developed a generic technology to industry? Or is it merely a specific technology for few companies?
Technological connections The extents to which the technology is applicable for many products. The technological connection is high if there are many technological applications Extendibility The potential of further technology developments based on the research results Soundness of scientific
principles
Is there any fundamental scientific problem? Is the scientific base sufficient for further technological development?
Quality of proposal Quality of the research proposal, including clear and measurable goals, feasible approach, good planning of resources/manpower, rational scheduling, solutions to problems
Capability of research team The capability of the research team, especially the team leader and the key technical staff
Safety and pollution concerns Concerns about public safety and pollution during the lifetime of the product, from project execution, commercial production to product consumption. The performance score is high when the concern is low
Success rate of commercia-lization
The probability of the success in technology transfer, product development and commercialization
Intensity of competition The intensity of market competition of the targeted products
Favourable environments The macroeconomic policy for the project, such as regulations, infrastructures, capital markets, etc.
Availability of complementary assets
The capability of firms to absorb and internalize the technology developed, and then to commercialize it
Timing Is it now the right timing to conduct this project?
Table 2. The preference structure of each group toward the evaluation objectives.
Objectives GA IN RI Average Economic benefits 0.188 0.134 0.167 0.164 (0.082, 0.437) (0.063, 0.468) (0.100, 0.596) (0.084, 0.512) Social benefits 0.137 0.056 0.055 0.082 (0.070, 0.513) (0.065, 1.155) (0.027, 0.493) (0.067, 0.816) Competitiveness 0.174 0.196 0.237 0.204 (0.069, 0.401) (0.071, 0.363) (0.109, 0.461) (0.087, 0.429) Relevance 0.147 0.181 0.177 0.169 (0.057, 0.390) (0.087, 0.482) (0.084, 0.476) (0.076, 0.453) Feasibility 0.148 0.210 0.206 0.189 (0.057, 0.389) (0.087, 0.415) (0.101, 0.492) (0.086, 0.458) Success Rate 0.207 0.222 0.157 0.194 (0.096, 0.464) (0.115, 0.520) (0.069, 0.438) (0.095, 0.491) Note: Figures in under parentheses show the standard deviation and coefficient of variation respectively.
(3) For experts from research organizations, the highest criteria scores are ‘proprietary tech-nology’ (0.082), ‘innovativeness of idea’ (0.079), ‘capabilities of research team’ (0.076), ‘advancement of technology’ (0.075) and ‘extendibility’ (0.071). This reflects that researchers put more emphasis upon technol-ogy than others.
Finally, a t-test is used to determine the differences in weights among the three groups with a minimum probability of 0.90. Assuming that the interest group population has a normal distribution, the results are summarized as follows:
(1) Regarding the objectives, the weight of the GA group towards ‘social benefits’ is sig-nificantly higher than the other two groups, and the weight of the IN group towards ‘feasibility’ is higher than the weight of the GA group.
(2) Regarding the criteria, weights of the GA group toward ‘benefits for human life’, and ‘improvements on QESIS’ are higher than the other two groups. The weight of RI group toward ‘proprietary technology’ is higher than the other two groups, and the weight of RI group toward ‘capability of research team’ is higher than the weight of GA group.
Table 3. The preference structure of each group toward the evaluation criteria.
Criteria GA IN RI Average
Market scope of application 0.052 0.026 0.050 0.043 (0.038, 0.733) (0.019, 0.736) (0.127, 2.522) (0.044, 1.028) Growth of target products 0.052 0.030 0.034 0.038
(0.021, 0.399) (0.025, 0.838) (0.066, 1.962) (0.023, 0.609) Added-value of target products 0.056 0.048 0.057 0.054
(0.029, 0.521) (0.028, 0.589) (0.039, 0.698) (0.031, 0.572) Relatedness of industry 0.029 0.031 0.028 0.029
(0.015, 0.533) (0.023, 0.764) (0.048, 1.707) (0.025, 0.858) Improvements on QESIS 0.047 0.021 0.019 0.029
(0.031, 0.671) (0.0250, 1.188) (0.114, 5.979) (0.027, 0.935) Concatenation with S&T policy 0.026 0.019 0.015 0.019
(0.019, 0.732) (0.025, 1.395) (0.038, 2.495) (0.018, 0.889) Benefits for human life 0.064 0.016 0.021 0.033
(0.03, 0.607) (0.026, 1.638) (0.114, 5.514) (0.035, 1.047) Innovativeness of idea 0.078 0.095 0.079 0.084 (0.0480, 0.611) (0.045, 0.476) (0.146, 1.834) (0.048, 0.576) Advancement of technology 0.052 0.052 0.075 0.060 (0.054, 1.035) (0.042, 0.811) (0.098, 1.308) (0.059, 0.989) Proprietary technology 0.043 0.049 0.082 0.059 (0.016, 0.361) (0.046, 0.929) (0.084, 0.476) (0.038, 0.648) Generics or specific 0.043 0.059 0.049 0.050 (0.017, 0.408) (0.051, 0.877) (0.082, 1.691) (0.039, 0.788) Technology connection 0.042 0.052 0.058 0.059 (0.023, 0.539) (0.028, 0.526) (0.128, 2.229) (0.032, 0.622) Extendibility 0.062 0.070 0.071 0.068 (0.039, 0.642) (0.055, 0.782) (0.057, 0.817) (0.048, 0.705) Soundness of scientific principles 0.024 0.025 0.034 0.028
(0.022, 0.934) (0.017, 0.667) (0.0271, 0.808) (0.021, 0.771)
Quality of proposal 0.036 0.045 0.054 0.045
(0.017, 0.488) (0.013, 0.292) (0.048, 0.884) (0.029, 0.660) Capability of research team 0.048 0.073 0.076 0.066
(0.024, 0.494) (0.056, 0.774) (0.083, 1.084) (0.046, 0.706) Safety and pollution concerns 0.040 0.067 0.042 0.049
(0.021, 0.526) (0.055, 0.822) (0.110, 2.633) (0.0431, 0.868) Intensity of competition 0.038 0.057 0.038 0.044
(0.035, 0.931) (0.026, 0.458) (0.109, 2.860) (0.028, 0.633) Favorable environment 0.057 0.054 0.034 0.048
(0.037, 0.652) (0.053, 0.988) (0.0243, 0.716) (0.040, 0.852) Availability of complementary assets 0.054 0.043 0.035 0.044
(0.028, 0.518) (0.0196, 0.458) (0.047, 1.331) (0.0286, 0.655)
Timing 0.059 0.067 0.049 0.058
These reflect the fact that different emphases among groups do exist. Government and acade-mia care more about social benefits than the other two groups, the researchers are more concerned about intellectual properties, and the experts from industry emphasize the importance of feasibility.
4.3. Illustrative examples
The industrial Technology Research Institute (ITRI), a non-profit research organization lo-cated in Hsinchu, was established in 1973. ITRI has been demonstrated to be successful both in fostering the development of new emerging industries, and in enhancing the level of sophis-tication of existing technology. It has more than 6000 professional employees, of whom nearly 60% have an advanced degree. Certain technol-ogies are presently the preferred targets, such as semiconductors, computers, communications, opto-electronics, biotechnology, microelectro-nics, micromechamicroelectro-nics, advanced materials and fine chemicals.
Previously, ITRI organized a Technology Advisory Committee (TAC) for consultation and review of the preferred projects. The review-ing processes are described as follows: first, research proposals are initiated at the division level. Second, initial reviews by peers coupled with intensive discussion are conducted within the same research laboratory. These procedures vary among laboratories, and are usually informal with limited well-defined guidelines. Third, a proposal is then submitted to the headquarters of ITRI and reviewed. ITRI has organized five committees to review the proposals from labora-tories based on fields of specialization. Several procedures are written for administrative purpose
and some broad criteria like ‘technological impacts’ and ‘markets potential’ are observed in these documents. But there are no clearly stated evaluating criteria and weights. Evaluators from TAC and the top management team review the proposals by conference discussions. The review committee makes the final calls for proposals.
Union Chemical Laboratories (UCL), a la-boratory of ITRI, was selected as a research target for the purpose of comparison. UCL has its own documented review procedures and has practiced them since 1992. The evaluation process was a mix of intuitive method and weighting-average method. Evaluators first reviewed and scored each project according to some arbitrary criteria/weights. Then the projects were ranked by a weighting-average method. The internal experts also provided an intuitive ranking of their preferences. Finally, the laboratory director selected the projects based on their ranks as well as his strategic judgments. We chose two from seven frontier technology programmes proposed by Union Chemical Laboratories in the year 2000. The two were chosen because they repre-sented two major types of ITRI’s projects. The first one was the ‘Metallocene Based Polymer Program (MBPP)’ which consisted of six highly related projects. The second was the ‘Specialty Chemicals Program’ (SCP), in which there were six unrelated projects. Both of the programmes had been evaluated according to UCL’s proce-dures and guidelines in 1999. We invited nine experts to repeat the evaluation by using the proposed method and then compared with previous results. All the experts were experienced, chemistry-related technical experts, many of whom were recognized as distinguished research-ers in ITRI. Each evaluator was asked to assign their subjective weights in Triangular Fuzzy
Table 4. Performance scores of MBPP projects toward each objective.
Objectives/ Projects Economic benefits Social benefits
Competi-tiveness Relevance Feasibility Success Rate Overall BNP Project A 12.8 5.7 13.5 12.9 13.9 11.7 70.5 (11.2,12.8,14.4) (4.7,5.7,6.7) (11.1,13.4,15.9) (11.1,12.8,14.6) (11.8,14.0,16.0) (9.3,11.7,14.0) (59.4,70.5,81.7) Project B 11.1 5.2 12.0 12.2 12.4 12.1 65.0 (8.9,11.1,13.1) (4.1,5.3,6.4) (9.2,12.0,14.7) (10.4,12.2,14.1) (9.7,12.4,14.9) (9.6,12.1,14.6) (52.0,65.0,77.8) Project C 13.5 5.8 14.3 12.5 14.1 12.5 72.7 (12.0,13.5,14.8) (4.8,5.8,6.8) (12.0,14.2,16.6) (10.9,12.5,14.2) (12.0,14.2,16.2) (10.2,12.6,14.7) (61.9,72.9,83.4) Project D 11.3 5.8 13.8 11.8 12.8 12.3 67.9 (9.4,11.4,13.3) (4.8,5.8,6.8) (11.1,13.8,16.4) (9.9,11.8,13.7) (10.5,12.8,15.1) (9.8,12.4,14.7) (55.5,68.2,79.3) Project E 13.0 5.8 11.9 12.0 13.1 14.1 69.9 (11.5,13.0,14.5) (4.8,5.8,6.8) (8.9,11.9,14.9) (10.3,12.0,13.7) (10.8,13.1,15.4) (12.0,14.2,16.1) (58.3,70.1,81.4) Project F 12.8 5.7 13.1 12.8 12.7 13.2 70.2 (11.1,12.7,14.4) (4.7,5.7,6.7) (10.2,13.2,16.0) (11.2,12.8,14.4) (10.1,12.8,15.1) (10.8,13.3,15.5) (58.1,70.4,82.2)
Numbers to the five levels of the linguistic variables. They then evaluated each of the 12 projects according to the criteria of hierarchy. Overall performance scores according to the evaluating hierarchy and their objectives/weights are shown in Tables 4 and 5. All the projects under review were assigned a nominal letter in order to retain confidentiality.
Tables 6 and 7 show the comparison of ranking results between the Fuzzy MCDM method and the current approaches. The ranking order of SCP by fuzzy multiple criteria approach is H I -K - G - J - L. The orders are identical in the lowest ranks J and L, but a slight difference is observed among the three methods for the remaining data. The ranking order of MBPP by fuzzy multiple criteria approach, C - A - F - E - D - B, differs from the rankings by intuitive judgment and weight average methods, which are A - C - E - B - F - D and A - C - E - F - B - D respectively. If we divide the project ranking into three groups, the ranking orders of the three methods are similar, with A and C being the highest, E and F in the middle, and B and D being the lowest.
Further analysis revealed that: (1) the ranking orders of the evaluation are not as meaningful as we suspected. Generally speaking, highly ranked projects, such as project C in MBPP’s and project H in SCP’s, can then be accepted with little further consideration. Similarly, lowly ranked projects such as project F in the SCP programme can be rejected with no further consideration. However, projects with intermediate scores need further investigation. (2) The individual perfor-mance scores toward each criterion are even more meaningful than the ranking orders generated by this approach, because the former can provide an abundance of information for further revision
and improvement. For example, a low score project may be accepted as a delayed project for further evaluation if it scores highly in ‘economic benefits’ but low in ‘innovativeness of idea’. Some deficiencies in the research proposal can be improved by the efforts of the researchers, since it is common practice to ask the researchers to reinforce the proposals after review. (3) The performance scores based on current proposals may change dramatically in a short period for reasons such as a breakthrough in competing technology, innovative ideas, and departure of key researchers. Therefore, a low-ranking project does not mean a ‘poor’ project and it may vary over time. However, the ranking orders are still useful for official competition purposes.
4.4. Feasibility of application
In order to investigate the applicability of the method and to identify possible difficulties, we organized two workshops to discuss it and interviewed some members of the project-funding office. The authors presented the results of our study to 22 participants after the evaluation was completed, and asked them to respond to a semi-structured questionnaire and discuss the feasi-bility of the method. The results are summarized as follows:
(1) Regarding the application of the fuzzy multi-ple criteria approach to frontier technology R&D projects, 72% of the respondents answered ‘feasible’. Some of them said that the method appeared reasonable but much too complicated. However, it became accep-table after our explanation that there was no need for evaluators to understand the detailed mathematical manipulation. Representatives Table 5. Performance scores of SCP projects toward each objective.
Objectives/ Projects Economic benefits Social benefits
Competi-tiveness Relevance Feasibility Success Rate Overall BNP Project G 12.4 5.5 13.3 12.5 12.6 13.1 69.5 (10.7,12.4,14.1) (4.5,5.5,6.5) (10.7,13.3,15.9) (10.8,12.5,14.3) (10.3,12.7,14.9) (10.9,13.1,15.4) (57.9,69.5,81.2) Project H 13.1 6.1 14.1 13.0 13.5 13.1 72.9 (11.5,13.1,14.6) (5.2,6.2,7.0) (11.6,14.1,16.6) (11.3,13.0,14.7) (11.4,13.5,15.5) (10.9,13.1,15.4) (62.0,73.0,83.8) Project I 12.7 6.2 13.8 11.7 13.7 12.7 70.6 (11.0,12.7,14.4) (5.3,6.2,7.0) (11.4,13.7,16.2) (9.9,11.7,13.6) (11.5,13.7,15.7) (10.3,12.6,15.1) (59.5,70.6,82.0) Project J 13.3 5.7 13.2 11.3 12.2 12.9 68.5 (11.7,13.3,14.8) (4.6,5.7,6.7) (10.5,13.2,16.0) (9.2,11.2,13.4) (9.8,12.3,14.6) (10.5,12.8,15.2) (56.3,68.5,87.8) Project K 12.6 5.6 12.7 12.2 12.8 13.7 69.6 (10.9,12.7,14.3) (4.6,5.6,6.6) (9.8,12.7,15.6) (10.3,12.1,14.1) (10.4,12.9,15.2) (11.6,13.7,15.8) (57.5,69.7,81.6) Project F 11.5 4.8 9.9 11.2 11.9 12.5 61.8 (9.6,11.6,13.4) (3.7,4.8,5.9) (6.9,9.9,12.8) (9.3,11.2,13.1) (9.3,11.9,14.4) (10.1,12.5,15.0) (48.8,62.0,74.6)
from research institutes thought that it could be one of the selection methods, but it should not be the only method. The reason is that mathematics cannot yield good results if the inputs or the measurements are incorrect due to the uncertainty associated with the devel-opments of frontier technology. The intuitive judgment of experts without clear or pre-determined criteria can be still a good approach.
(2) Compared with current approaches, 60% of the respondents agree that Fuzzy MCDM method was ‘better’. Several advantages of the fuzzy multiple criteria approach are mentioned:
(a) A clearly stated procedure with well defined criteria and weights can improve the visibility of evaluation. It is good for a ‘fair’ perception and for the purposes of proposal preparation. (b) It can effectively solve the disparity between ‘profound knowledge required for evaluation’ and ‘different expectation from various inter-est groups’. The results of project selection will be more acceptable to different interest groups because such an approach uses criteria and weights that are determined by
the stakeholders. Then evaluation by techni-cal experts can improve the deficiency of poor knowledge of interest groups.
(c) The fuzzy set theory is more applicable when dealing with the linguistic variables of ex-perts’ judgments and the ambiguities of frontier technology.
We also presented this method and its results to members of the project-funding office. They were impressed with the methodology in three aspects: first, it provides a ‘visible’ process and a ‘measurable’ result of evaluation. Several mem-bers stressed that it is more important to have an ‘open and fair’ process than a good outcome in selecting government-sponsored projects. Second, the different interests of stakeholders can be easily integrated by the AHP. Different weights of criteria reflect their desired emphasis. Third, this approach can promote communication among the key players. Fourth, this approach can solve the disparity between the profound knowledge required for evaluation and the different expectations of various interest groups. However, their major criticism of fuzzy MCDM is again the mathematical complexity.
5. Concluding discussion
This paper presents a fuzzy multiple criteria approach for the selection of government-spon-sored R&D projects and reports the experience in applying it at a national research institute in Taiwan. This approach includes: (1) using the analytical hierarchy process (AHP) method to integrate various expectations from different interest groups into evaluating objectives/criteria, (2) the group-decision method by technical experts based on predetermined objectives/criter-ia, and (3) the fuzzy approach in scoring the subjective judgments of the experts.
Three characteristics of the government-spon-sored NFTP projects make the selection more difficult: First, the nature of government-sup-ported projects, for which various expectations from different interest groups must be taken into account. The interested parties include govern-ment, research organizations, and industry. Sec-ond, the relatively weak knowledge base in frontier technology areas that exists in developing countries like Taiwan. Third, the vagueness of new frontier technology, its technical uncertain-ties, market risks, lack of hard data, and lack of qualified evaluators are part of the reasons why Table 6. Ranking comparison of Metallocene based
Polymer Program (MBPP). Current methods Project Fuzzy MCDM BNP (ranking) Intuitive ranking Weight average method (ranking) A 70.5(2) 1 81.4(1) B 65.0(6) 4 74.9(5) C 72.7(1) 2 78.9(2) D 67.9(5) 6 71.6(6) E 69.9(4) 3 77.4(3) F 70.2(3) 5 75.6(4)
Table 7. Ranking comparison of Specialty Chemicals Program (SCP). Current Methods Project Fuzzy MCDM BNP (ranking) Intuitive ranking Weight average method (ranking) G 69.5(4) 2 82.9(1) H 72.9(1) 1 82.4(2) I 70.6(2) 3 80.9(4) J 68.5(5) 5 79.9(5) K 69.6(3) 4 82.4(3) L 61.8(6) 6 77.0(6)
evaluation usually proceeds subjectively and intuitively. Therefore, the major purpose of this paper is to solve the disparity between the profound knowledge required for evaluation and the different expectations from various interest groups. Our study reveals that differences of weights toward each criterion among different interest groups do exist. Government and acade-mia care more about social benefits, researchers are more concerned about intellectual properties, and experts from industry emphasize the impor-tance of feasibility.
The method we present in this paper has several advantages: (1) this method can resolve the disparity between the profound knowledge required for evaluation and the different expecta-tion from various interest groups. Evaluaexpecta-tion of new frontier technology projects requires profi-cient and highly specialized knowledge from scarce experts, and the evaluation results from technical experts cannot reflect the various expectations of different groups. The approach of using a review committee organized by representatives from the different interest groups is also not optimal because most of these representatives do not possess the necessary knowledge for evaluation of the projects. On the other hand, the current approach in ITRI by using TAC, which is superior to the former approach in terms of the quality of technical judgment, is usually not able to reflect the different desire of stakeholders. The fuzzy multi-ple criteria approach we proposed here can successfully resolve this disparity. (2) It provides an ‘open and fair’ process that is essential for building consensus and commitment. A clearly stated procedure with well defined criteria and weights can improve the visibility of evaluation and is good for a ‘fair’ perception. (3) This method provides an abundance of information for further improvement. Many of the deficiencies in the research proposal can be improved by the efforts of researchers, since it is a common practice to ask researchers to reinforce the proposals after review. Raising the value of individual projects would be of much greater value than simply ranking projects and reallocat-ing resources. One of the benefits of this approach is not in discovering the best project to fund but in stimulating researchers to develop better projects. This finding is consistent with the view of some authors that a good selection method should be also a ‘decision aid’ to facilitate communication and to improve the quality of projects (Bordley, 1998; Brussion, 1980;
Henrik-sen and Traynor, 1999; Schmidt and Freeland, 1992).
While we believe that the method we presented provides value, several issues remain: (1) the model did not consider all possible interactions among criteria and projects; (2) the result of our study did not answer the question about the applicability of methodology. We are actively seeking a new situation to apply this model. (3) mathematical complexity is still the major barrier to a wider acceptance. Further improvements including decision-aiding software are recom-mended.
References
Alidi, A. (1996) Use of the analytical hierarchy process to measure the initial viability of industrial projects. International Journal of Project Management, 14, 4, 205–208.
Baker, N.R. (1974) R&D project selection models: an assessment. IEEE Transactions on Engineering Man-agement, EM-21, 2, 165–171.
Baker, N.R. and Freeland, J. (1975) Recent advances in R&D benefit measurement and project selection methods. Management Science, 21, 1164–1175. Baker, N.R. and Pound, W.H. (1964) R and D project
selection: where we stand. IEEE Transactions on Engineering Management, EM-11, 124–134.
Bellman, R.E. and Zadeh, L.A. (1970) Decision-making in a fuzzy environment. Management Science, 17, 4, 141–164.
Bergman, M. and Mark, G. (2002) Exploring the relationship between project selection and require-ments analysis: an empirical study of the new millennium program. Proceedings of the IEEE Joint Conference on Requirement Engineering.
Bordley, B.F. (1998) R&D project selection versus R&D project generation. IEEE Transactions on Engineering Management, 45, 4, 407–413.
Brenner, M.S. (1994) Practical R&D project prioritiza-tion. Research Technology Management, 27, 5, 38– 42.
Brussion, N. (1980) The function of project evaluation. R&D management, 10, 2, 61–65.
Chang, P.L. and Hsu, C.W. (1997) A project manage-ment system (PMS) for research institutes applying for government R&D contracts in Taiwan. Interna-tional Journal of Project Management, 15, 3, 165– 172.
Chen, S.J. and Hwang, C.L. (1992) Fuzzy Multiple Attribute Decision-Making: Methods and Applica-tions, Berlin: Springer.
Coffin, M.A. and Taylor III, B.W. (1996) Multiple criteria R&D project selection and scheduling using fuzzy logic. Computers and Operations Research, 123, 3, 207–220.
Danila, N. (1989) Strategic evaluation and selection of R&D project. R&D Management, 19, 1, 47–62. Dias, O.P. Jr. (1988) The R&D project selection
problem with fuzzy coefficients. Fuzzy Sets and Systems, 26, 3, 299–316.
Dubis, D. and Prade, H. (1978) Operations on fuzzy numbers. International Journal of Systems Science, 9, 613–626.
Dubis, D. and Prade, H. (1980) Fuzzy Sets and Systems: Theory and Applications, Academic. Fahrni, P. and Spa¨tig, M. (1990) An
application-oriented guides to R&D project selection and evaluation methods. R&D Management, 20, 155–171. Gaynor, H. (1990) Selecting projects. Research
Tech-nology Management, 33, 4, 43–45.
Henriksen, A.D. and Traynor, A.J. (1999) A practical R&D project selection scoring tool. IEEE Transac-tions on Engineering Management, 46, 2, 158–170. Islei, G.G., Lockett, B.C. and Stratford, M. (1991) A
decision supports system using judgmental modeling: a case of R&D in the pharmaceutical industry. IEEE Transactions on Engineering Management, 38, 3, 202–209.
Liberatore, M.J. and Titus, G.B. (1983) The practice of management science in R&D project management. Management Science, 29, 962–974.
Liberatore, M.J. (1987) An extension of the analytic hierarchy process for industrial R&D projects selection and resource allocation. IEEE Transactions on Engineering Management, 34, 4, 12–18.
Lockett, G., Hetherington, B., Yallup, P., Stratford, M. and Cox, B. (1986) Modeling a research portfolio using AHP: a group decision process. R&D Manage-ment, 16, 2, 151–160.
Martino, J.P. (1995) R&D Project Selection, New York: Wiley.
Mathews, J.A. (2002) The origins and dynamics of Taiwan’s R&D consortia. Research Policy, 31, 633–651.
Meade, L.M. and Presley, A. (2002) R&D project selection using the analytical network process. IEEE Transactions on Engineering Management, 49, 1, 59–66.
Saaty, T.L. (1980) The Analytic Hierarchy Process, New York: McGraw Hill.
Schmidt, R.L. and Freeland, J.R. (1992) Recent progress in modeling R&D project-selection pro-cesses. IEEE Transactions on Engineering Manage-ment, 39, 2, 189–201.
Steele, L.W. (1988) What we’ve learned: selecting R&D programs and objectives. Research Technology Man-agement, 31, 2, 17–36.
Teng, J.Y. and Tzeng, G.H. (1996) Fuzzy multicriteria ranking of urban transportation investment alter-native. Transportation Planning and Technology, 20, 1, 15–31.
Tsaur, S.H., Tzeng, G.H. and Wang, K.C. (1997) Evaluating tourist risks from fuzzy perspectives. Annals of Tourism Research, 24, 4, 796–812. Vonortas, N.S. and Hertzfeld, H.R. (1998) Research
and development project selection in the public sector. Journal of Policy Analysis and Management, 17, 4, 621–638.
Zadeh, L.A. (1965) Fuzzy sets. Information and Control, 8, 3, 338–353.
Zhau, R. and Goving, R. (1991) Algebraic character-istics of extended fuzzy numbers. Information Science, 54, 103–130.
