An effective evaluation model and improvement analysis for national park websites: A case study of Taiwan

(1)

Case Study

An effective evaluation model and improvement analysis for national park websites: A case study of Taiwan

Wen-Hsien Tsai

^a^,^*

, Wen-Chin Chou

^b

, Chien-Wen Lai

^c

aDepartment of Business Administration, National Central University, Jhongli, Taoyuan 320, Taiwan

bDepartment of Accounting, Yu Da University, Chaochiao, Miaoli 361, Taiwan

cDepartment of Accounting and Information, Asia University, Wufeng, Taichung 413, Taiwan

a r t i c l e i n f o

Article history:

Received 22 January 2009 Accepted 23 January 2010

Keywords:

National park Website evaluation

Decision-Making Trial and Evaluation Laboratory (DEMATEL)

Analytic Network Process (ANP) VlseKriterijumska Optimizacija I Kompromisno Resenje (VIKOR)

a b s t r a c t

Taiwanese National Park Headquarters employ advanced information technologies to attract travellers, researchers, and other visitors, but it is likely that not all Headquarters have clear knowledge about how successful their websites are. This study proposes an effective model for evaluating national park websites. The model ﬁrst applies the Decision-Making Trial and Evaluation Laboratory (DEMATEL) to cope with the interdependencies between evaluation criteria. Next, it uses the Analytic Network Process (ANP) to compute weights for each criterion. Finally, it uses the VlseKriterijumska Optimizacija I Kom- promisno Resenje (VIKOR) to rank Taiwanese national park websites. Overall, the results show that each national park website must be improved in order to become a high quality website. Furthermore, the weight-variance analysis suggests managerial actions based on two-dimensional maps for improving website quality. Therefore, this study not only provides a comprehensive and systematic approach that quantitatively measures a website’s overall performance, but also contributes to practical applications in terms of providing worthwhile recommendations for building an ideal website.

1. Introduction

National parks are repositories of unique natural scenery, cultural assets, and historic resources that are both popular and signiﬁcant as tourism sites (Ma, Ryan, & Bao, 2009). Taiwanese national parks provide important recreational and tourism- related resources for domestic visitors and international tourists (CCPA, 2008). Following current global trends, and in response to the request of the United Nations, the Taiwanese government chose 2002 as ‘‘The Year of Ecotourism in Taiwan,’’ and had promoted ecotourism as a way to attract more foreigners to visit national parks and understand ecology of these places (Tao, Eagles, & Smith, 2004). Speciﬁcally, the visitor arrivals to Taiwanese national parks grew 20% within 4 years, from 15,118,078 in 2002 to 18,203,609 in 2006, and the annual growth rates were 2%, 5%, 7%, 10%, respectively (CCPA, 2008). Moreover, the Taiwanese Tourism Bureau had announced 2008 and 2009 as

‘‘Tour Taiwan Years’’ to attract many international tourists to the island. The director of the Taiwanese Tourism Bureau invited

international tourists to experience B&B accommodations in a mountain village, along with healthful hiking and eco- explorations. International tourists were also invited to tackle some more challenging outdoor activities, such as climbing the highest mountain (Yushan) in Northeast Asia (Lai, 2008).

The Taiwanese Government has established seven national parks to respond to the growing demand for tourism opportunities and environmental protection. The ﬁrst Taiwanese national park was established in 1984, and six more parks have been added since that time. Of the seven national parks currently operating in Taiwan, Yushan, Yangmingshan, Taroko, Shei-pa are located in mountainous regions, Kenting and Dongsha Marine near beaches or ocean/marine areas, and Kinmen at a major cultural/historic site. Along with beautiful scenery, they provide shelter to unique animal and plant life, including insects, ﬁsh, and birds. The natural reserves actually form miniature ecosystems that not only provide a protected environment but also offer various opportunities for recreational activities, environmental education, and academic research. Here, visitors can escape from their hectic lives in the city and enjoy a serene environment (Tourism Bureau, 2008a).

Seven National Park Headquarters were set up to take charge of the substantial resource conservation education and construction management activities in order to promote the parks’ sustainable development in a planned manner. The National Park

*Corresponding author. Tel.: þ886 3 425 0860; fax: þ886 3 422 2891.

E-mail address:whtsai@mgt.ncu.edu.tw(W.-H. Tsai).

Contents lists available atScienceDirect

Tourism Management

j o u r n a l h o m e p a g e : w w w . e l s e v i e r . c o m / l o c a t e / t o u r m a n

doi:10.1016/j.tourman.2010.01.016

(2)

Headquarters have given much effort to managing and preserving (or restoring) a natural environment (CCPA, 2008). Through the Headquarters’ continuing efforts to develop and promote national parks, Taiwanese national parks are now included in the list of world parks (Hsu, 2007).

Due to the popularity of internet, the National Park Headquar- ters are able to deliver ecological conservation information, environmental education, and tourism information through their websites. At present, there is a multilingual website for each national park, which provides international visitors with a good deal of information for their travel needs. Computer-mediated tour information, such as traffic, activities, and tour-package information can increase awareness, interest, and the likelihood of visiting a specific travel destination (Ho & Chou-Yen, 2003). Moreover, the content of tourism destination websites is particularly important because it directly leads to the creation and communication of the destination’s perceived image, which produces a virtual experience for visitors (Doolin, Burgess, & Cooper, 2002). The National Park Headquarters do not, however, understand how successful their websites are or how many gaps should be filled between the status quo and an ideal website. In other words, how much effort must the Headquarters put into improving or enhancing website quality in order to achieve their desired/aspired ends? This raises the critical issue of how the national park managers can effectively measure their websites’ performance. An evaluation is required to answer this question, and a sound methodology is the key to effective measurement. No comprehensive mechanism for systematically assessing the various elements of a website, however, has yet been introduced. Moreover, existing methods do not offer enough insights for practitioners to determine whether their websites meet ideal levels in terms of design and content.

In this vein, this study aims to build an effective model for evaluating national park websites, and then prioritize improvement actions in order to best allocate available resources. To achieve this goal, an effective model is proposed that combines the Decision-Making Trial and Evaluation Laboratory (DEMATEL) method (Fontela & Gabus, 1976), the Analytic Network Process (ANP) method (Saaty, 1996), and the modiﬁed VIKOR (VlseKriter- ijumska Optimizacija I Kompromisno Resenje in Serbian, or Multi- criteria Optimization and Compromise Solution) method (Opricovic, 1998) for evaluating and ranking national park websites.

This model ﬁrst applies the DEMATEL method to deal with the interdependence between evaluation criteria and convert the criteria’s cause and effect relations into a visual structural map. Next, ANP is employed to determine the relative weights of evaluation criteria. The ANP, as a multiple criteria decision-making (MCDM) method, can be used to systematically deal with network-like decision problems. Finally, the VIKOR method is used to assess and rank the websites of seven national parks in Taiwan. The VIKOR method introduces an aggregating function, which then represents the site’s distance from an ideal solution. This ranking index is an aggregation of all criteria, including the relative importance of criteria and a balance between total and individual satisfaction (Opricovic & Tzeng, 2004). One of its advantages is to take into account the lowest performance rating with respect to a speciﬁed criterion. Additionally, a weight-variance analysis (WVA), based on the concept of importance-performance analysis (IPA) (Martilla &

James, 1977), is proposed to identify weakness areas most in need of urgent remedial action. The ﬁndings of this study can help National Park Headquarters form a clear picture of their websites’

quality level. Furthermore, this study will be a valuable contribu- tion for enhancing website design and for achieving desired quality levels. Hence, this combined four-phase method represents an effective tool for evaluating national park websites and for priori- tizing improvement actions.

2. The concepts of website evaluation 2.1. The criteria of website evaluation

Website evaluation measures have been proposed in various contexts in recent years; researchers in this area struggle to determine important factors for evaluating online service and marketing.

Zeithaml (2002)argued that electronic service quality (e-SQ) has seven dimensions that form two scales: 1) a core e-SQ scale that includes efficiency, fulfilment, reliability, and privacy; and 2) a recovery e-SQ scale that includes responsiveness, compensation, and contact. After a rigorous scale-development process was applied to the e-SQ,Parasuraman, Zeithaml, and Malhotra (2005) developed the E-S-QUAL as a measure of core service quality and the E-RecS-QUAL as a subscale for problem resolution. Subse- quently, Kim, Kim, and Lennon (2006)proposed a modified E-S- QUAL model expected to capture extensive service attributes available on apparel retail websites. This model includes nine dimensions: the six dimensions from the original E-S-QUAL (efficiency, fulfilment, system availability, privacy, responsiveness, and contact) along with three additional dimensions (personalization, information, and graphic styles).Miranda-Gonza´lez and Ban˜ egil- Palacios (2004) proposed a detailed Web Assessment Index, which focuses on four categories: accessibility, speed, navigability, and content.Cao, Zhang, and Seydel (2005)used factor analysis to capture the quality of three e-trade websites, which includes system quality, information quality, service quality, and attractiveness.Ho and Lee (2007) also used factor analysis to develop an e-travel service quality scale, which includes five core components: information quality, security, website functionality, customer relationships, and responsiveness. Barnes and Vidgen (2002)developed a scale to measure an organisation’s e-commerce offering; the scale provides an index of a site’s quality and includes five factors:

usability, design, information, trust, and empathy.

Closely linked to the concept of website quality is the notion of usability. Usability has been defined and measured in many different ways (Agarwal & Venkatesh, 2002). Nielsen (1993) proposed that usability has five attributes: learnability, efficiency, memorability, low error rate or easy error recovery, and satisfaction. He also suggested that ‘‘usability is a quality attribute that assesses how easy user interfaces are to use’’ (Nielsen, 2003). The International Organisation for Standardization (ISO) defined usability as ‘‘the extent in which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use’’ (ISO, 1998).Nielsen (2000) extended information system design principles for Web and suggested four parameters for usability: (1) navigation, (2) response time, (3) credibility, and (4) content.Agarwal and Venkatesh (2002) utilized the Microsoft Usability Guidelines to define website usability through five categories (content, ease of use, made-for- the-medium, promotion, and emotion), while Palmer (2002) defined usability based on five basis design elements extracted from usability and design as well as media richness literature (download delay, navigability, content, interactivity, and responsiveness).Au Yeung and Law (2006), Bai, Law, and Wen (2008)and Qi, Law, and Buhalis (2008) categorized usability into language usability, layout and graphics, information architecture usability, user interface and navigation, and general usability.Hassan and Li (2005)identified web usability as screen appearance consistency, accessibility, navigation, media use, interactivity, and content.Kim and Kim (2008) identified four usability criteria, which includes usefulness, effectiveness, satisfaction and supportiveness. On the other hand,Tarafdar and Zhang (2005)suggested that information content, ease of navigation, download delay, and website availability positively influence website usability.Pearson, Pearson, and

(3)

Green (2007) investigated the relative importance of ﬁve web usability criteria (navigation, customization and personalization, download speed, accessibility, and ease of use). They indicated

‘‘ease of use’’ is the most important criterion.

Based on a detailed review of previous literature, the relevant criteria for assessing national park websites are summarized as Table 1.

2.2. The methods of website evaluation

Many attempts have been made to address website evaluation for different organisational sectors and website categories. In particular, much literature focused on evaluation of travel-related websites (Baloglu & Pekcan, 2006; Doolin et al., 2002; Law, 2007;

Lee & Kozar, 2006; Wan, 2002), hospital and government websites (Bilsel, Buÿu¨ko¨zkan, & Ruan, 2006; Buÿu¨ko¨zkan & Ruan, 2007), and education and e-learning websites (Buÿu¨ko¨zkan, Ruan, &

Feyziogg˜lu, 2007; Kasli & Avcikurt, 2008; Shee & Wang, 2008).

There is no universally accepted method or technique for website evaluation. Various assessment techniques have been employed to evaluate websites using subjective approaches based on individual preferences, such as the Analytic Hierarchy Process (AHP) (e.g.,Lee & Kozar, 2006; Shee & Wang, 2008), the Technique for Order Preference by Similarity to an Ideal Solution (TOPSIS) (e.g.,Buÿu¨ko¨zkan & Ruan, 2007; Law, 2007), the Pref- erence Ranking Organisation Method for Enrichment Evaluation (PROMETHEE) (e.g., Bilsel et al., 2006), and the VIKOR (e.g., Buÿu¨ko¨zkan et al., 2007).Bilsel et al. (2006)used the AHP and Fuzzy PROMETHEE ranking methods to evaluate website quality for nine hospitals according to seven e-service quality dimensions (tangibles, reliability, responsiveness, confidence, empathy, quality of information, and integration of communication).Lee and Kozar (2006) applied AHP to evaluate four online electronics and four online travel websites by adopting DeLone and McLean’s IS success model. AHP can only obtain relative weights for criteria and alternatives; it cannot compute gaps between the status quo and an ideal point of an alternative.Buÿu¨ko¨zkan et al.

(2007)used the Fuzzy VIKOR method to evaluate 21 e-learning websites according to seven criteria (right and understandable content, complete content, personalization, security, navigation, interactivity, and user interface). They neglected interrelationships between these seven criteria when determining their weights.Bu¨yu¨ko¨zkan and Ruan (2007)used Fuzzy AHP and Fuzzy TOPSIS to rank 13 Turkish government websites according to six e-service quality dimensions. TOPSIS, however, has some limitations. According to Wang, Luo, and Hua (2007), TOPSIS’

closeness coefﬁcient values do not reﬂect the superiority or inferiority of alternatives and therefore cannot be used for ranking purposes.

Other methods include evaluating websites using content analysis.Cai, Card, and Cole (2004)andBaloglu and Pekcan (2006) utilized content analysis to analyze the websites of 20 US tour operators and 139 hotels, respectively, using a measurement variable of yes/no (1/0).Kasli and Avcikurt (2008)also used content analysis to examine the websites of 132 tourism departments at universities. The shortcoming of binary variables is that they cannot express the performance of each criterion (i.e., the quality of various features). In addition,Wan (2002)used content analysis to evaluate the websites of 30 tourist hotels and 39 tour wholesalers using a ﬁve-point rating scale. He took into account the performance on each criterion, but the relative importance of various criteria was neglected.

Table 2summarizes the disadvantages of previous studies of website evaluation. As shown inTable 2, previous studies have failed to provide a comprehensive and systematic approach that quantitatively measures a website’s overall performance, and their research methodologies must be improved. Therefore, this study proposes an effective evaluation model that combines DEMATEL, ANP, and a modiﬁed VIKOR to assess national park websites in terms of website quality. The proposed model overcomes the drawbacks of prior studies and offers enough insights for National Park Headquarters to accurately measure the current level of their websites according to critical criteria.

It is worth noting that usability evaluation techniques have been developed and incorporated into the design and development of websites. These techniques were used to evaluate interfaces for the purpose of identifying problems in order to improve usability of the interfaces (Ahmed, 2008). Several studies used usability techniques to evaluate websites’ usability (e.g.,Ahmed, 2008; Aitta, Kaleva, &

Kortelainen, 2008). These studies involved a focus on the interface and discovered different kinds of usability problems; however, our proposed model used MCDM techniques to evaluate and rank various websites.

Table 1

Website quality evaluation criteria.

Criterion Deﬁnition

Navigability This criterion measures how easy it is to navigate around the site, how easy it is to return to the home page of the site, how easy it is to ﬁnd relevant information (Miranda-Gonza´lez & Ban˜ egil-Palacios, 2004), how many links are required to get from one point in a site to another, and what search tools the site provides (Smith, 2001).

Speed This criterion refers to quick connection and delivery, minimal use of large graphics and bright colours, easy access to links (Bilsel et al., 2006), and website loading speed (Smith, 2001).

Links This criterion refers to availability of links to other government organisations (Bu¨yu¨ko¨zkan & Ruan, 2007), different national parks, eco-protection, tourism and travel, and other related websites.

Relevancy This criterion includes relevant depth and scope and completeness of information (Lee & Kozar, 2006).

Different parts of the website should be designed to meet the needs of different group of visitors (Cao et al., 2005), such as travellers, researchers, students, and local citizens.

Richness This criterion refers to detailed level and scope of information content. That is, formations contained on the website are rich in content (Bilsel et al., 2006).

Currency This criterion refers to up-to-date information.

Last update/review dates are a critical way of notifying users of the currency of content (Lee & Kozar, 2006; Smith, 2001).

Attractiveness This criterion consists of whether web pages are fun to read and help visitor promote their excitement, such as through graphics, online games, cartoons, screensavers, software downloads, and Q&As (Cao et al., 2005; Huizingh, 2000; Miranda-Gonza´lez &

Ban˜ egil-Palacios, 2004).

Security This criterion deals with how a website proves to be trustworthy for customers (Ho & Lee, 2007). A conﬁdent website should assure the secrecy of its users’ personal and private data as well as prevent the content of a message from being tampered with

(Bu¨yu¨ko¨zkan et al., 2007; Chu, 2001).

Personalization This criterion includes an individualized interface, effective one-to-one information, and customized service (Lee & Kozar, 2006). Customized content of the website can provide a user with the relevant and up-to-date information that will address his speciﬁc needs (Ho & Lee, 2007).

Responsiveness This criterion deals with the provision of information on FAQs and prompts assistance for solving problems (Ahn, Ryu, & Han, 2007; Ho & Lee, 2007). Various service functions, such as complaint management systems (Lee & Kozar, 2006), should be provided.

(4)

3. The four-phase method for website evaluation

To effectively evaluate national park websites and suggest improvement actions, a combined four-phase method is proposed.

This combined four-phase method is a novel hybrid method which integrated DEMATEL, ANP, the modiﬁed VIKOR, and WVA. Previous studies have applied the integrated AHP-VIKOR method to solve different problems such as restaurant location selection (Tzeng, Teng, Chen, & Opricovic, 2002), environmental management (Tzeng, Tsaur, Laiw, & Opricovic, 2002), and public transportation (Tzeng, Lin, & Opricovic, 2005). AHP is based on the additive concept along with the independence assumption, but each individual criterion is not always completely independent in reality (Wu & Lee, 2007). Therefore, instead of adopting AHP, we combined DEMATEL method and ANP method to treat interdependence between criteria.

The weights obtained through ANP were combined with the modi- ﬁed VIKOR method to compute the comprehensive performance variance rate of each website. VIKOR can only obtain the ranking of websites; it cannot identify weakness areas most in need of urgent remedial action. Therefore, we applied WVA to suggest improvement actions.

The evaluation process includes four phases: (1) applying the DEMATEL method; (2) applying the ANP method; (3) applying the modiﬁed VIKOR method; and (4) employing the WVA method.

Accordingly, an overview of the evaluation process for national park websites is shown in Fig. 1. The details of each phase are described below.

First, we apply the DEMATEL method prior to the ANP procedure in order to improve the procedure for dealing with the interrelationships between criteria. Previous ANP studies (e.g.,Lee & Kim, 2000; Lin, Tsai, Shiang, Kuo, & Tsai, 2009; Wey & Wu, 2007) coped with interdependence of criteria through group discussion and did not quantify the strength of interdependence between criteria. Therefore, DEMATEL method is used to quantify complex relationships between criteria and to convert these relationships into an inﬂuence-relation-map (IRM) (Tsai & Chou, 2009; Tsai, Chou, & Hsu, 2009).

Second, we apply the ANP method to build the network structure for evaluating websites by using the IRM, and then to calculate the weights for each criterion. The ANP method is a comprehensive decision-making technique that has the capability of dealing with complex interrelationships between attributes and decision levels, while the AHP model’s decision-making structure uses straight- forward hierarchical relationships among decision levels

(Chang, Wu, Lin, & Lin, 2007; Tuzkaya, O¨ nu¨t, Tuzkaya, & Gu¨lsu¨n, 2008). Therefore, instead of adopting the commonly used AHP method for solving these types of problems, we used an ANP-based model for calculating the weights for each criterion.

Then, we modify the traditional VIKOR method regarding the measure of closeness to an ideal alternative. The modiﬁed VIKOR offers more rational formulas for computing the gaps between the status quo and the ideal point for websites (i.e., unimproved gaps).

Finally, we employ WVA to identify those criteria of website quality that are most in need of urgent remedial action. WVA offers an understandable guide to assist National Park Headquarters in organising limited resources toward improvement actions.

The combined four-phase method not only can adequately cope with the interdependence between criteria but also can suggest improvement actions to reduce the gaps between the status quo and the ideal point for websites. The following describes in more detail the four methods used in this study.

3.1. The DEMATEL method

The DEMATEL method was devised for a Science and Human Affairs Program by the Battelle Memorial Institute of Geneva Table 2

Summary of the disadvantages of previous studies of website evaluation.

References Methods Evaluated websites Disadvantages

Lee and Kozar (2006) AHP Online electronics and online travel websites

They used AHP to rank websites. AHP can only obtain relative weights for criteria and alternatives; it cannot compute gaps between the status quo and an ideal point for websites.

Bilsel et al. (2006) AHP and Fuzzy PROMETHEE

Hospital websites They used PROMETHEE II to rank websites. PROMETHEE II can only provide a complete pre-ordering through a comparison of net outranking ﬂows; it cannot compute gaps between the status quo and an ideal point.

Bu¨yu¨ko¨zkan et al. (2007)

Fuzzy VIKOR E-learning websites They ignored interrelationships between the seven criteria when determining their weights.

Bu¨yu¨ko¨zkan and Ruan (2007)

Fuzzy AHP and Fuzzy TOPSIS

Government websites They used TOPSIS to rank websites. TOPSIS’ closeness coefﬁcient values do not reﬂect the superiority or inferiority of alternatives and therefore cannot be used for ranking purposes (Wang et al., 2007).

Cai et al. (2004) Content analysis Tour operator websites They used a measurement variable of yes/no (1/0). The shortcoming of binary variables is that they cannot express the performance of each criterion (i.e., the quality of various features).

Baloglu and Pekcan (2006)

Content analysis Hotel websites They used a measurement variable of yes/no (1/0). The shortcoming of binary variables is that they cannot express the performance of each criterion (i.e., the quality of various features).

Kasli and Avcikurt (2008)

Content analysis University websites They used a measurement variable of yes/no (1/0). The shortcoming of binary variables is that they cannot express the performance of each criterion (i.e., the quality of various features).

Wan (2002) Content analysis Tourist hotel and tour wholesaler websites

He took into account the performance on each criterion, but the relative importance of various criteria was ignored.

Calculating the weights for each criterion

Identifying weakness areas for managerial actions Analyzing the interrelationships

between evaluation criteria DEMATEL

ANP

VIKOR

WVA

Finding the ranking of websites Determining evaluation criteria for measuring the

performance of national park websites

Fig. 1. The evaluation process for national park websites.

(5)

between 1972 and 1976. DEMATEL converts the mutual relationships between criteria’s causes and effects into an intelligible structural model for the system (Tzeng, Chiang, & Li, 2007). It has been successfully applied in a wide range of situations, such as R&D projects (Lin & Wu, 2008), e-learning evaluations (Tzeng et al., 2007), airline safety measurements (Liou, Tzeng, & Chang, 2007), social responsibility programs (Tsai & Hsu, 2008; Tsai, Hsu, Chen, Lin, & Chen, 2009), socially responsible investments (Tsai, Chou, et al., 2009), sustainable management (Tsai & Chou, 2009), and IT projects sourcing strategy mix decision (Tsai, Leu, Liu, Lin, & Shaw, 2010). This method progresses as follows:

Step 1: Producing the initial direct-relation matrix. The evaluators who are selected in terms of professional knowledge and skill are asked to indicate the degree of direct influence of each criterion over others. To ensure objectivity, there are at least three evaluators to execute the evaluation (Van der Merwe & Bekker, 2003). The pair-wise comparison scale can be broken down into five levels including ‘‘no influence (0),’’ ‘‘low influence (1),’’ ‘‘medium influence (2),’’

‘‘high inﬂuence (3),’’ and ‘‘very high inﬂuence (4)’’

(Chiu, Chen, Tzeng, & Shyu, 2006; Liou et al., 2007). The result of this comparison produces an initial direct- relation matrix. The initial direct-relation matrix B is an n n matrix, where b_ijis denoted as the degree to which the ith criterion affects the jth criterion. It is represented by Eq.(1). Accordingly, all principal diagonal elements b_ij of matrix B are set to be zero. Assume each time only changes a parameter in matrix B. There are 5 ½nðn 1Þ

possible mixes. Excel software can be applied to aid in the calculations of sensitivity analyses. For example, when b_ij are changed from 2 (medium influence) to 3 (high influence), the number of elements whose influence levels in the total-relation matrix are greater the threshold value will increase. In other words, each time the value of a parameter increases, some new interrelationships will be added to the IRM.

B ¼ 2 66 66 4

b₁₁ / b_1j / b_1n

« « «

b_i1 / b_ij / b_in

« « «

b_n1 / b_nj / bnn

3 77 77

5 (1)

Step 2: Calculating the normalized direct-relation matrix. The normalized direct-relation matrix can be obtained through formulas (2)–(4), where all principal diagonal elements are equal to zero (Chiu et al., 2006).

X ¼ B=r (2)

r ¼ Max 0

@ max

1in

Xⁿ

j¼1

jb_ijj; max

1jn

Xⁿ

i¼1

jb_ijj 1

A; i;j˛f1;2;3;.;ng (3)

i/NlimXⁱ ¼ ½0_nn; where X ¼ x_ij

nn; 0 x_ij<1 (4)

Step 3: Computing the total-relation matrix. Once the normalized direct-relation matrix X has been obtained, a continuous decrease in problems’ indirect effects along the powers of the matrix X, e.g., X², X³,., X^N, guarantees convergent solutions to the matrix inversion. The total-relation matrix T can be derived by using formula(5), where I is denoted as the identity matrix (Chiu et al., 2006; Liou et al., 2007).

T ¼ X þ X²þ X³þ . ¼ X^N

i ¼ 1

Xⁱ ¼ XðI XÞ¹ (5)

Step 4: Computing the values of influence and relation. Using the values of D R and D þ R, where D is the sum of columns and R is the sum of rows in matrix T, levels of influence on others and levels of relationships with others are defined as shown in formulas(6)–(8) (Hori & Shimizu, 1999; Wu &

Lee, 2007). Some criteria having positive values of D R and thus greatly inﬂuence other criteria. These criteria are called dispatchers; others having negative values of D R and thus are greatly inﬂuenced by other criteria. These are called receivers. The value of D þ R indicates the degree of relationship between each criterion with other criteria.

Criteria having higher values of D þ R have stronger relationships with other criteria, while those having lower values of D þ R have less of a relationship with others (Seyed-Hosseini, Safaei, & Asgharpour, 2006).

T ¼ t_ij

nn i; j˛f1; 2; 3; .; ng (6)

D ¼ 2 4Xⁿ

j ¼ 1

t_ij 3 5

n1

¼ ½t_i,_n1 (7)

R ¼

"

Xⁿ

i ¼ 1

t_ij

#t

1n

¼ t_:j

n1 (8)

where superscript t denotes transposition.

Step 5: Setting a threshold value to obtain the IRM. If all the information from matrix T converts to the IRM, the map will be too complex to show necessary information for decision-making. To obtain an appropriate IRM, the decision-maker must set a threshold value for the influence level. Only some elements, whose influence levels in matrix T are greater than the threshold value, can be chosen and converted into the IRM. The threshold value is decided by experts or decision-makers through discussions (Tzeng et al., 2007). If the threshold value is too low, the map will be too complex to show the necessary information for decision-making. In contrast, if the threshold value is too high, many factors will be presented as independent factors, without showing the relationships with other factors. Each time the threshold value increases, some relationships will be removed from the map (Tzeng et al., 2007). An example based on a total-relation matrix Têxis shown as formula(9)and inFig. 2. An appropriate threshold value is necessary to obtain a suitable IRM as well as adequate information for further analysis and

(6)

decision-making (Li & Tzeng, 2009). Finally, the IRM is acquired by mapping the dataset (D þ R, D R), where the horizontal axis is D þ R, and the vertical axis is D R (Wu & Lee, 2007)

The positive signiﬁcant value of D R represents that the criterion affects other criteria much more than the other criteria affected it, implying it should be a priority for improvement. In managerial implications, the results of the DEMATEL can provide some insights for organisations to improve their performance based on the most powerful criterion that greatly inﬂuences the performance of other criteria (Tsai, Chou, et al., 2009). Therefore, the supplemental value of getting the information regarding interdependency exceeds the added complexity of applying DEMATEL. It is suitable to apply DEMATEL prior to the ANP procedure.

3.2. The ANP method

The ANP, developed by Thomas L. Saaty, provides a means to input judgments. The ANP also provides measurements to derive ratio scale priorities for the distribution of inﬂuence between factors and groups of factors in the decision (Saaty, 2003). ANP has been successfully applied to many practical decision-making problems, such as ERP software selection (Ayag˘ & O¨ zdem_Ir, 2007), logistics service provider selection (Jharkharia & Shankar, 2007), digital video recorder systems evaluation (Chang et al., 2007), undesirable facilities location selection (Tuzkaya et al., 2008), and social responsibility programs (Tsai & Hsu, 2008; Tsai, Hsu, Chen, et al., 2009). According to Saaty (2001), the ANP comprises six main steps:

Step 1: Conducting pair-wise comparisons on the elements using Saaty’s nine-point scale (Saaty, 2001). The scale ranges from equal importance (one) to extreme importance (nine).

Step 2: Computing relative importance weights (eigenvectors) for each element and testing the consistency ratio (CR). If the CR is greater than 0.1, the result is not consistent, and the pair-wise comparison is performed again.

Step 3: Placing the results of these computations within the supermatrix (unweighted). The supermatrix concept resembles a Markov chain process. To obtain global priorities in a system with interdependent inﬂuences, the local priority vectors are added to the appropriate columns of a matrix, which is known as a supermatrix.

Step 4: Conducting pair-wise comparisons on the clusters.

Step 5: Weighting the blocks of the unweighted supermatrix by the corresponding cluster priorities, such that the result is column-stochastic (weighted supermatrix).

Step 6: Raising the weighted supermatrix to limiting powers until the weights converge and remain stable (limit supermatrix).

3.3. The modiﬁed VIKOR method

VIKOR has been introduced as an applicable technique for implementation within MCDM (Opricovic, 1998; Opricovic & Tzeng, 2002, 2003, 2004; Tsai, Hsu, & Lin, 2009; Tzeng et al., 2005; Tzeng, Teng, et al., 2002; Tzeng, Tsaur, et al., 2002); VIKOR focuses on ranking and selecting (from a set of alternatives) in the presence of conﬂicting criteria (Opricovic & Tzeng, 2007). This method considers two distance measurements, S_j and Q_j, based on an aggregating function ðLp metricÞ in the compromising programming method in order to provide information about utility and regret; the best alternative has the maximum group utility for decision-makers and ensures the least regret (Opricovic & Tzeng, 2004, 2007). Thus, the VIKOR method can provide measurements of determining the aggregate relative distance between the perceptive performances and the ideal performances of alternatives (Tsai, Hsu, & Lin, 2009).

The VIKOR method includes the following steps:

Step 1: Determining the maximum f_i^*and the minimum f_ivalues of all criterion functions, i ¼ 1, 2,., n. If the criterion i represents a beneﬁt, then f_i^* ¼ max

j f_ij;f_i ¼ min

j f_ij. Natu- rally, a candidate having scores ðf₁^*;f₂^*; .; f_n^*Þ would be a positive-ideal candidate, whereas a candidate having scores ðf₁;f₂; .; fnÞ would be a negative-ideal candidate.

Step 2: Computing the values of S_jand Q_j. These values represent group utility and individual regret for the alternative a_j, respectively, with the relations

S_j¼ L^p¼1_j ¼Xⁿ

i¼1

w_ih

f_i^* f_ij.

f_i^* f_ii

; for j ¼ 1;.;m (10)

Q_j ¼ L^p¼N_j ¼ max

i

nw_ih

f_i^* f_ij.

f_i^* f_iii

¼ 1; 2; .; no

; for j ¼ 1; .; m (11)

G3 G1

G2

G4

Threshold value (p)= 1.0 G3

G1

G2

G4

Threshold value (p)= 1.0

G3 G1

G2

G4

G1

G2

G4

G3 G1

G2

G4

G1

G2

G4

Fig. 2. Inﬂuence-relation-maps based on the different threshold values.

T^ex =

G1 G2 G3 G4 G1

G2 (9) G3 G4

1.108 1.501 1.575 1.712 1.317 1.188 1.609 1.695 1.042 1.123 1.056 1.472 0.893 0.962 1.048 0.976

(7)

where the weights of the criteria ðwiÞ are introduced in order to express the relative importance of criteria as computed by the ANP method.

Step 3: Computing the aggregate value ðR_jÞ. Its formula is:

R_j¼vh

S_jS^*.

SS^*i

þð1vÞh

Q_jQ^*.

QQ^*i

;for j

¼1;.;m (12)

where S^*¼min

j S_j;S¼max

j S_j;Q^*¼min

j Q_j;and Q¼max

j Q_j; v is introduced as a weight for the strategy of maximizing group utility, whereas 1v is the weight of the individual regret.

Step 4: Ranking alternatives by sorting each Sj, Qj, and Rjvalues in an increasing order. The result is a set of three ranking lists denoted by S(), Q() and R().

Step 5: Proposing the alternative a⁰, which is ﬁrst ranked by the measure minfR_jjj ¼ 1; 2; .; mg as a single optimal solution. The alternative must satisfy two conditions as follows:

N1. The alternative a⁰has an acceptable advantage; in other words, Rða⁰⁰Þ Rða⁰Þ h where h ¼ 1/(m 1) and m is the number of alternatives (h ¼ 0.25 if m 4).

N2. The alternative a⁰is stable within the decision-making process;

in other words, it is also the best ranked in S() or/and Q().

If one of the above conditions is not satisﬁed, then a set of compromise solutions is proposed, which consists of:

Alternatives a⁰and a⁰⁰, if only the condition N2 is not satisﬁed, or

Alternatives a⁰;a⁰⁰; .; a^ðkÞ, if the condition N1 is not satisﬁed;

and a^ðkÞ is determined by the relation Rða^ðkÞÞ Rða⁰Þzh (the positions of these alternatives are ‘‘in closeness’’).

In the traditional VIKOR method, the values of S^*, S, Q^*, and Qcome from the candidate alternatives; there are not real positive-ideal values or negative-ideal values for all criteria.

Therefore, following Ou Yang, Shieh, Leu, and Tzeng (2009), the positive-ideal value S^*would be zero; the negative-ideal value S would be equal to one; the positive-ideal value Q^*would be zero, and the negative-ideal value Qwould be equal to one. In addition, Eq.(13)is used instead of the traditional Qj(i.e., the weights ðw_iÞ were removed from Eq.(11)). The modiﬁed Q_j^modand R^mod_j index are listed as follows:

Q_j^mod¼max

i

hf_i^*f_ij.

f_i^*f_ii¼1;2;.;ni

;for j¼1;.;m (13)

R^mod_j ¼ vS_jþ ð1 vÞQ_j^mod (14)

It is important to note that the value of h should be changed to h^mod ¼ ðmax

j R^mod_j min

j R^mod_j Þ=ðm 1Þ in the modiﬁed VIKOR method. Other conditions are the same as those with the traditional VIKOR method.

3.4. The weight-variance analysis

As mentioned above, the ranking index of the modiﬁed VIKOR method represents a compound measurement of S_jand Q_j^mod. The S_jvalue is comprised of the ANP weight ðw_iÞ of the ith criterion and the performance variance rate ððf_i^* f_ijÞ=ðf_i^* f_iÞÞ of the ith criterion of the jth alternative. The Q_j^modvalue represents the jth

alternative with respect to the ith criterion calculated by the highest performance variance rate. To describe the components of S_jand Q_j^modin detail, the WVA (which is based on the IPA concept ofMartilla & James, 1977) is proposed, except that the ‘‘performance variance rate’’ replaces the ‘‘performance’’ component and the ‘‘ANP weight’’ is substituted for the ‘‘importance’’ component.

WVA provides a more ﬁne-grained analysis focusing on the gaps between an actual website and an ideal website according to particular evaluation criteria. The performance variance rate is then plotted against the ANP weight to provide a graphic repre- sentation of which evaluation criteria are most in need of improvement. The ‘‘ANP weight’’ constitutes the vertical axis (y- axis) and ‘‘performance variance rate’’ constitutes the horizontal axis (x-axis) of a coordinate diagram, which is labelled the weight- variance map (WVM). We then followAbalo, Varela, and Manzano (2007) in using a partition that combined four quadrant and diagonal-based schemes for the WVM, enlarging the top right quadrant so that the new region occupied the whole of the zone above the iso-rating line. Evaluation criteria located above the iso- rating line are a higher priority for improvement (Chang & Yang, 2008). Finally, the map is divided into four zones, as shown in Fig. 3. The following are recommendations for each of the four zones:

(1) Concentrate here (Zone 1): Evaluation criteria in this zone are perceived to be important for evaluators; however, the performance variance rate is high in this zone. Criteria falling in this zone constitute the top priority for remedial action, and the necessity of improvement is proportional to the horizontal distance from the iso-rating line (Chang & Yang, 2008).

(2) Keep up the good work (Zone 2): This zone indicates not only those criteria deemed important to evaluators, but also a low performance variance rate. Criteria in this zone require careful monitoring to ensure that low variance rate levels are maintained.

(3) Redeploy resources (Zone 3): This zone contains criteria of low importance, and the performance variance rate is also relatively low. Managers should not be overly concerned

Zone 1

(Concentrate Here)

Performance Variance Rate

thgieW PNA

Zone 2 (Keep Up the Good Work)

Zone 3

(Redeploy Resources)

Zone 4 (Low Priority) All criteria above the iso-rating line are in need of improvement

Low High woLhgiH

Fig. 3. The weight-variance map.

(8)

about criteria in Zone 3; consideration should be given to the possibility of redeploying resources to remedial action in Zone 1.

(4) Low priority (Zone 4): Evaluation criteria in Zone 4 are rated as having low importance and a high performance variance rate. It is therefore not necessary to focus additional effort or resources to criteria in this zone.

The IPA method, a facile and visible analysis, reveals strengths and weaknesses of attributes for identifying areas in need of improvement (Chen & Chang, 2005; Skok, Kophamel, & Richardson, 2001). This study modiﬁed the original IPA and transformed the components of the VIKOR index into the WVA. This modiﬁcation provides a powerful tool for guiding the prioritization of improvement actions.

In sum, the similarity of these four methods is that the data come from subjective judgments of evaluators; the difference of them is that each method has its own capability for solving a speciﬁc problem. First, the DEMATEL method not only can deal with interrelationships of criteria but also can integrate them into a visible structural map. Second, the ANP method not only can build the network structure by using the IRM but also can calculate the weights for each criterion. Next, the weights

obtained through ANP are combined with the modiﬁed VIKOR method to compute performance variance rates between the status quo and the ideal point for websites. Finally, WVA can integrate ANP weights and performance variance rates into a two-axis map and suggest improvement actions. However, these four methods have their own advantages and disadvantages, as shown inTable 3.

4. Evaluation of the national park websites

To describe the proposed model clearly, a case study of national park websites was conducted in order to demonstrate the efﬁcacy of this model for assessing website quality. The research objects in this study were all national park websites in Taiwan. There are seven national parks in Taiwan: Kenting, Yushan, Yangmingshan, Taroko, Shei-pa, Kinmen, and Dongsha Marine. Their websites are indicated as A1, A2, A3, A4, A5, A6, and A7, respectively. We invited 16 experts to express their opinions in November 2008. Nine of them were from various industries and the remaining ones were from academics and research institutes. These experts are professionals who have been designing websites or studying on various projects about e-government and e-business for a certain time; therefore, Table 3

Comparison of the DEMATEL, ANP, modiﬁed VIKOR and WVA methods.

Methods Advantages Disadvantages

DEMATEL It is an effective tool for analyzing structure and relationships between components of a system (Tzeng et al., 2007).

It is time-consuming to choose a consistent threshold value, especially if there are too many experts’ opinions to aggregate at the same time (Li & Tzeng, 2009).

ANP It allows for more complex relationship among the decision levels and attributes as it does not require a strict hierarchical structure (Ravi, Shankar, & Tiwari, 2005).

The pair-wise comparison of criteria under consideration can only be subjectively performed, and their accuracy of the results depends on the evaluator’s expertise knowledge in the area concerned (Ravi et al., 2005).

Modiﬁed VIKOR It determines a compromise solution that could be accepted by the decision-makers because it provides a maximum group utility for the ‘‘majority’’, and a minimum of individual regret for the ‘‘opponent’’

(Tzeng, Teng, et al., 2002; Tzeng, Tsaur, et al., 2002; Tzeng et al., 2005).

It uses a ﬁxed common numbers of criteria for all projects.

However, it cannot aggregate the unimproved gaps according to the particular criteria for each project/aspect (Ou Yang et al., 2009).

WVA (Modiﬁed IPA) It is a relatively quick and inexpensive method for identifying areas in need of improvement by using a simple and visible map (Skok et al., 2001).

It clearly sacriﬁces depth and is unlikely to provide the detailed insights found from in-depth interviews (Skok et al., 2001).

Navigability (C1)

Speed (C2) Links (C3)

Relevancy (C4)

Richness (C5)

Currency (C6) Attractiveness (C7)

Evaluation of national park websites

Kenting Website

(A1)

Yushan Website (A2)

Dongsha Website (A7) Yangmingshan

Website (A3)

Taroko Website (A4)

Shei-pa Website

(A5)

Kinmen Website (A6)

Goal

Criteria

Alternatives

Fig. 4. A standard network for the case study.

(9)

their answers to questionnaires can appropriately reﬂect the status quo of each national park website.

4.1. Calculation of the weights of evaluation criteria

First, the experts indicated that the presence of too many criteria would make the website evaluation process difﬁcult and complex.

As mentioned in Section2, ten relevant criteria were identiﬁed as key factors for website quality. To ensure that these potential criteria were valid measures for evaluating national park websites, the experts examined the status quo of seven websites beforehand and then used the cut-off value method with the nine-point scale to screen for proper criteria. Based on the survey results of these important criteria, seven criteria were constructed as very important to the success of the websites, with a mean value exceeding 6.0 (i.e., cut-off value) for each criterion. The seven criteria were noted as navigability (C₁), speed (C₂), links (C₃), relevancy (C₄), richness (C5), currency (C6), and attractiveness (C7). The remaining three criteria (i.e., security, personalization, responsiveness) were excluded because they were considered to be relatively unimpor- tant for the evaluation of national park websites (with mean scores of 3.1, 2.5, and 4.2, respectively).

In the recent years, the issue of accessibility has become a concern for many countries, including the Taiwanese government.

The Executive Yuan (i.e., the highest administrative agency in Taiwan) has established the Accessible Web Development Guide- lines (AWDG) based on the Web Content Accessibility Guideline 1.0 of Web Accessibility Initiative from World Wide Web Consortium (RDEC, 2008). The seven national park websites have obtained ‘‘web accessibility conformance seals’’ from the Executive Yuan and they continuously comply with AWDG. Therefore, accessibility was not included in the proposed model, as this criterion performed well in all seven websites. In addition, the ‘‘Alternatives’’ cluster consisted of seven national park websites.Fig. 4shows the standard network for this case study.

Second, in order to determine the relationship structure among the seven evaluation criteria, a zero–four DEMATEL scale was used for assessment purposes. This scale was designated as five levels: the score of 0 (No influence), 1 (Low influence), 2 (Medium influence), 3 (High influence), and 4 (Very high influence), respectively. It can help

experts apply the pair-wise comparisons to model a mathematical relationship between criteria. Once the relationships between criteria had been measured by the experts, the initial direct-relation matrix could then be obtained (Table 4). Based on an initial direct- relation matrix, a normalized direct-relation matrix was obtained using the DEMATEL formula. Next, the total-relation matrix (including D, R, D þ R and D R) was identified (Table 5). After deciding on a threshold value (p ¼ 0.500) based on discussions with the evaluators, the relationships between criteria were identified by mapping the D þ R and D R dataset (Fig. 5). As shown inFig. 5, evaluation criteria were visually divided into a dispatcher group, which included speed (C2), links (C3), relevancy (C4), and currency (C₆), while the receiver group included navigability (C₁), richness (C5), and attractiveness (C7). The link (C3), with the highest value of D R, was called the master dispatcher. A good national park website provides all necessary Web links covering tourist spots, visitors’ services, transportation, accommodation, and ecological conservation information; therefore, links (C3) affected each of the other six criteria: navigability (C1), speed (C2), relevancy (C4), richness (C₅), currency (C₆), and attractiveness (C₇). The IRM also indicated that links (C3) and currency (C6) strongly influenced other criteria for improving website quality. In contrast, attractiveness (C7), with the lowest value of D R, was the master receiver and was affected by each of the other six criteria. These results support Cao et al. (2005), who suggest that an attractive website begins with good content. Moreover, attractiveness (C₇), with the highest value of D þ R, had the most relationships with other criteria. Finally, the results of the IRM informed the ANP in building the network relationship structure.

Phase 2 applied the ANP method to calculate a weight for each criterion. The experts responded to the questionnaire through a series of pair-wise comparisons with Saaty’s one-nine scale, comparing the relative importance of one element over another.

After computing the results of experts’ assessments, the consistency ratio values were less than the acceptable threshold value (i.e., CR < 0.1) and the displayed eigenvectors were appropriate to enter into the supermatrix M (Table 6). The supermatrix did not need to be weighted since every column summed to one and the limit supermatrix could be computed directly. Finally, the supermatrix was raised to limiting powers of M^Lto capture all interactions and obtain Table 4

The initial direct-relation matrix.

Navigability (C1) Speed (C2) Links (C3) Relevancy (C4) Richness (C5) Currency (C6) Attractiveness (C7)

Navigability (C1) 0.000 2.063 1.688 2.063 2.375 2.063 2.875

Speed (C2) 3.188 0.000 1.750 1.563 1.813 2.000 2.938

Links (C3) 2.375 2.000 0.000 2.500 2.938 1.938 2.563

Relevancy (C4) 2.625 1.750 1.688 0.000 2.813 2.250 2.813

Richness (C5) 2.813 2.625 1.625 2.313 0.000 1.875 3.000

Currency (C6) 2.688 2.250 2.250 2.688 2.563 0.000 3.188

Attractiveness (C7) 2.563 2.188 1.688 1.875 2.125 1.938 0.000

Table 5

The total-relation matrix.

Navigability (C1) Speed (C2) Links (C3) Relevancy (C4) Richness (C5) Currency (C6) Attractiveness (C7)

D D D R D R

Navigability (C1) 0.490 0.507 0.424 0.505 0.562 0.483 0.662 3.633 8.040 0.774

Speed (C2) 0.649 0.403 0.429 0.483 0.538 0.481 0.667 3.650 7.251 0.049

Links (C3) 0.651 0.538 0.363 0.558 0.625 0.508 0.691 3.934 6.952 0.916

Relevancy (C4) 0.649 0.516 0.443 0.422 0.608 0.512 0.689 3.839 7.426 0.252

Richness (C5) 0.666 0.561 0.445 0.543 0.473 0.501 0.705 3.894 7.875 0.087

Currency (C6) 0.707 0.583 0.506 0.600 0.646 0.440 0.763 4.245 7.628 0.862

Attractiveness (C7) 0.595 0.493 0.408 0.476 0.529 0.458 0.495 3.454 8.126 1.218

R 4.407 3.601 3.018 3.587 3.981 3.383 4.672

Note: The bold values represent values higher than the threshold value (p ¼ 0.500).

(10)

a steady-state outcome. The limit supermatrix is shown inTable 7.

The results of the limit supermatrix yielded (C₁, C₂, C₃, C₄, C₅, C₆, C7) ¼ (0.221, 0.113, 0.010, 0.125, 0.255, 0.080, 0.196). Ranked by the weights, the top-three evaluation criteria were richness (C5), navigability (C1), and attractiveness (C7). These three criteria were influenced by the other criteria since the value of D R is negative according to the computation of the DEMATEL. The results of the ANP showed that the receiver group in the DEMATEL analysis obtained higher weights. If the interrelated structure of criteria were neglected, the results of the AHP would yield (C1, C2, C3, C4, C5, C6, C7) ¼ (0.146, 0.115, 0.077, 0.156, 0.179, 0.154, 0.173). Table 8 presents the relative weights for the seven criteria based on the results of the ANP, as well as the relative weights as determined by the AHP. It is interesting to contrast these two methods, as their results are significantly different in terms of derived weights and the ranking order of various criteria. The contrasting outcomes indicate that interdependencies between criteria can affect real decision- making processes. The ANP has a powerful capacity to solve more complex decision problems and deliver more reliable results than does the AHP (Ayag˘ & O¨ zdem_Ir, 2007; Chang et al., 2007). Conse- quently, adopting a suitable method is important, as it influences

the accuracy of the evaluation results. In addition, our study solved the problem of the study ofWan (2002), which only used a ﬁve- point rating scale to assess websites but neglected the relative importance for each criterion.

4.2. Performance measurement of the websites

After ﬁnishing a series of pair-wise comparisons, the evaluators were asked to provide linguistic values for the seven criteria. In this study, linguistic values were used to design the evaluation questionnaire. These performance values, which were very good, good, median, poor, and very poor, were transformed by scaling them into the numbers 100, 75, 50, 25, 0, respectively. Our study took into account the performance on each criterion and solved the problems of the studies ofBaloglu and Pekcan (2006), Cai et al. (2004)and Kasli and Avcikurt (2008), which only used a measurement variable of yes/no.

The average assessed value, f_ij, for the jth alternative according to ith criterion was determined by the relation

0.0

-1.0 D− R (Influence)

D+R (Relation) 8.0

Navigability (C₁) (8.040, -0.774) Relevancy (C₄)

(7.426, 0.252) Speed (C2)

(7.251, 0.049)

Currency (C₆) (7.628, 0.862)

Richness (C₅) (7.875, -0.087) Links (C3)

(6.952, 0.916)

-1.5 -0.5 0.5 1.5

1.0