Chapter 6 Evaluation
6.3 Experiments and Results
國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
75
6.3 Experiments and Results
The experiments are divided broadly into two parts. The first part was conducted to find a set of design parameters that can optimize the performance of the mechanism.
The second is to evaluate the above-mentioned hypotheses.
6.3.1 Design parameters
In the mechanism, there are three key parameters (the number of metaphors, the number of Google queries and the similarity threshold for attractiveness analysis) that need to be finely tuned in order to ensure a near optimal system performance.
Following are the descriptions of the parameters and the experiment results.
6.3.1.1 The number of metaphors
One of the fundamental principles for partner selection is to find the partners who have the complimentary capabilities and resources, which are embodied in terms of the images in our system. Considering the system process, an initial step is to find the image gap between the goal and the status quo. The gap analysis results will suggest what are the missing elements required to complement by others. The identified partners should be able to fill this image gap.
It is worth noting that the system would not generate possible partner candidates until the new metaphors have been made. During partner generation process, the system attempts to metaphorize the partners to something and then tries to match the something to the real business partners. The gap images are supposed to be embedded in these metaphors in order to ensure the image gap can be filled.
However, there is one thing we should bear in mind. That is, one of the most troubling aspects of these metaphors generation is that it cannot be always to find the metaphors embedded all the gap image elements. Sometimes they are just hardly to
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
76
find. Furthermore, considering the efficiency of generating the final output, it is practically unacceptable to find the best metaphor because it may takes too much time.
Therefore, we settle for the second option that we seek a metaphor with the highest gap image coverage index under a limited number of metaphors generated. The gap image coverage index formula is showed in Figure 6.4.
gap image coverage index = the number of gap images embedded in the metaphor the number of gap images
Figure 6.4.gap image coverage index formula
In this experiment, what we would like to know is how many metaphors should be generated in order to keep the image coverage index and the system efficiency performance in acceptable level. The results are shown in the Figure 6.5. The system efficiency performance here refers to the execution time of the proposed mechanism and its value has been normalized (e.g., 1 indicates high performance and 0 indicates low performance). The results appeared to suggest that the number of metaphor should be set to 3 because it reaches the balanced state between system efficiency performance and the gap image rate; however, for the research purpose, we decide to set the number of metaphors to 8 since it would be better to have a higher gap coverage rate. When the system is put into practice, this parameter would be change to 3.
‧
Figure 6.5 The test run for deciding the number of metaphors required to ensure acceptable system efficiency performance
6.3.1.2 The number of Google queries
Google search ajax API is adopted when the system sends a query to Google. It’s quite a powerful tool and it’s convenient to use, but it only return up to 64 results at a time. We found that many of the returned results are often duplicated or useless. For the sake of getting the wanted results, performing queries several times are unavoidable. This experiment was then designed to understand how many times queries have to be sent in order to get the wanted words.
For example, while the system sends a query like “as * as a noun”, a sets of words that can replace the wildcard character would be returned. Only some of results are desirable because all we need are meaningful adjectives. We may also get the results like “as long as a noun...”. These kinds of results are those needed to be filtered out. The unwanted word list should be prepared to automatically do so.
Furthermore, assuming that we want to get 10 meaningful adjectives in this example, if the system finally gets 8 adjectives, we would say that we get a result with 80%
image element discovery rate. The simple formula of image element discovery rate is
0
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
78
provided as follows. The optimal is the result with the highest image discovery rate.
image discovery rate = the number of images we actually get the number of wanted images Figure 6.6.image discovery rate formula
The results are illustrated in Figure 6.7. The horizontal axis indicated the number of queries sent to Google while the vertical axis depicted the image discovery rate.
The results highlights that the number of Google queries should be set to 4 because this setting would make image element discovery rate be close to 100%. It’s already high enough. The system performance has been significant improved after the setting was adopted.
Figure 6.7 The number of Google queries
6.3.1.3 The image similarity threshold for attractiveness analysis
In the final module of the proposed mechanism, attractiveness analysis is performed to evaluate the power to attract people by the given image configuration of a new
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
79
alliance. This process involves the comparison of image models. Let’s assume there is a predicted image model of a new alliance. This new image model would be compared with each representative image model of customer image clusters and a image similarity score would be generated for each cluster. The image similarity score denotes the level of similarity between image models and its value will be between 0 and 1(e.g., 1 represents two image models have similar image configuration and 0 indicates they have nothing in common). For more details of image model similarity, please reference section 4.5 in chapter 4.
We also assume that customer image model represents their preference and behavior at some level. If an image model configuration of an alliance is really similar to an image model of a customer (i.e., high similarity score), we believe this customer will be attracted by the feeling presented by this new alliance. However, to what extent is the level of similarity that we think it higher enough to say that the customer will be attracted by a specific alliance? It is clearly that a similarity threshold is needed and its value is undefined for now. This experiment was set up to determine the value of similarity threshold.
There are two important considerations to properly set this parameter. The first consideration is that this parameter should be higher enough to reflect the idea that the similarity score is supposed to be high when the customers are attracted. That’s why the thresholds we tested started from 0.71. The second is that the thresholds can not to be set too high or too low. To explain this, consider the process of attractiveness analysis.
The attractiveness analysis is to calculate the percentage of customers who are attracted by a specific alliance in our platform. We determine whether the customers are attracted by the alliance according to the level of similarity between the image models of customers and the alliance. If the similarity threshold is set too high, then
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
80
no customer would be determined “being attracted”. For example, assume that there are three customer image models (C1, C2, C3) and an alliance image model (A1). The attractiveness analysis would start by computing the level of similarity of image models for each pair, that is, the level of similarity between C1 and A1, C2 and A1, and C3 and A3. Assume that the levels of similarity are 0.75, 0.82 and 0.87 respectively. If the similarity threshold is set to 0.9 (too high), then the attractiveness score becomes 0 because no customer can be attracted; conversely, if the similarity threshold is set to 0.72 ( too low ), then the attractiveness score becomes 1 because all customers can be attracted. In either case, there is a possible side effect – the attractiveness scores usually remain the same score (0 or 1) for different alliances.
Then, we are not able to know the level of attractiveness differences between alliances.
That makes the attractiveness analysis out of function.
According to the important considerations mentioned above, the goal of this experiment was designed to ensure the threshold setting can make the attractiveness score effective. Simply put, it can tell the differences of the attractiveness level between different cases. As observed in Figure 6.8, the vertical axis indicates the percentage of cases which have different attractiveness score by setting a given threshold and the horizontal one denotes the value of threshold. The results suggested that the threshold should be set to 0.84. About 90% cases have different attractiveness score.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
81
Figure 6.8 The similarity threshold for attractiveness analysis
After all the parameters are well settled, we can further validate the usefulness of the proposed mechanism and test the hypotheses.
6.3.2 Evaluate the hypotheses
In this subsection, the hypotheses are tested through the experiments in order to answer the research questions mentioned earlier. The experiment setting is summarized below. The parameters are set according the results of previous section 12 goals are also prepared for the experiments.
Table 6.1. The parameter setting for the experiments
Paramters values
the number of metaphors 8
the number of Google queries 4
the similarity threshold for attractiveness analysis
0.84
‧
Table 6.2. The testing goalss prepared for the experiments
Goals Input Meaning
1 Heaven the service experience is just like being in Heaven
2 fascinating game the service experience is just like playing fascinating game
3 shining fireworks the service experience is just like watching shining fireworks
4 romantic wedding the service experience is just like attending romantic wedding
5 childhood fantasy the service experience is just like fulfill childhood fantasy
6 magical show the service experience is just like watching magical show
7 nostalgic village the service experience is just like being in nostalgic village
8 blockbuster movie the service experience is just like watching blockbuster movie
9 Olympics the service experience is just like attending Olympics
10 Dreamy Island the service experience is just like being in Dreamy Island
11 Bali Island the service experience is just like being in Bali Island
12 cultural festival the service experience is just like attending cultural festival
13 Hawaii the service experience is just like being in Hawaii
14 Paris the service experience is just like being in Hawaii
6.3.2.1 The level of goal fulfillment
The first step to evaluate the proposed mechanism is to guarantee it indeed can improve the level of goal fulfillment. The followings are the relevant hypotheses.
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
83
Hypothesis1: The proposed mechanism can help SMEs to identify the partners who have great potential to jointly build the goal images.
Hypothesis 1-A: In the context of high image diversity, a SME would have more chances to improve the level of goal fulfilment through cooperating with others.
Hypothesis 1-B: In the context of low image diversity, a SME would have few chances to improve the level of goal fulfilment through cooperating with others.
Figure 6.9 The level of goal fulfillment under low level image diversity context
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7
goal 1 goal 2 goal 3 goal 4 goal 5 goal 6 goal 7 goal 8 goal 9 goal 10 goal 11 goal 12 goal 13 goal 14
The level of goal fulfillment
lower level image diversity context
Before cooperating with partners
After cooperating with partners
‧
Figure 6.10 The level of goal fulfillment under high level image diversity context
Figure 6.11 The level of improvement on goal fulfillment for each goal in two different context settings
In this experiment, business image models were randomly selected from business image classes as the target businesses who want to discover partners by metaphor-based partner recommendation mechanism.. Two different levels of image diversity contexts were formulated to test the impact of this factor on goal
The level of goal fulfillment improvement
low diversity context high diversity context
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
85
achievement activity. Specifically, there were five image business classes in low diversity context. Each business class represented one possible image type of businesses. Therefore, five business image models were separately selected from five business image classes in a random manner. We are able to understand the performance of goal fulfillment for each different business image types. After the target businesses were selected, we started to input 14 goals separately to gain 70 results (i.e., 5 target businesses x 14 different goals) and their fulfillment scores. This step was taken in both context settings, and then we’re able to examine the difference between them.
The complete results are shown in Figure 6.9, 6.10 and 6.11. In Figure 6.9 and 6.10, the solid line indicates the level of goal fulfillment after the target businesses cooperates with the partners our system recommended and the dotted line refers to the level of goal fulfillment before the target businesses cooperates with others. The goal fulfillment scores are averaged base on the different goal settings. According the formula of goal fulfillment score provided in section 4.5, the goal fulfillment scores in these two Figures will be between 0 and 1. The higher value means the higher level of goal fulfillment and vice versa. The detail of the goal fulfillment score and its rationales are provided in chapter 4, goal fulfillment analysis.
In general, as observed in these three Figures, the level of goal fulfillment can be improved through cooperating with the partners the system recommended, no matter in which context. This improvement is reflected in gaining higher goal fulfillment score. Figure 6.9 and 6.10 highlights the differences between cooperating with partners (i.e., the solid line) and doing business by the SME itself (i.e., the dotted line).
Cooperating with partners may have more chance to improve their images.
However, it has to be stressed that image diversity is a particularly important variable in considering the likelihood of improving the level of goal achievement. In
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
86
Figure 6.11, it demonstrates level of improvement on goal fulfillment for each goal in two different context settings. These results support the idea that the image diversity of a destination would impact on the level of difficulties in goal achievement. That is, in higher image diversity context, businesses are much easier to find the appropriate partners to collectively achieve the goal. In sum, hypotheses 1 (The proposed mechanism can help SMEs to identify the partners who have great potential to jointly build the goal images), 1-A (In the context of high image diversity, a SME would have more chances to improve the level of goal fulfilment through cooperating with others) and 1-B (In the context of low image diversity, a SME would have few chances to improve the level of goal fulfilment through cooperating with others) are supported by these results.
6.3.2.2 The level of uniqueness
Hypothesis 2: In the context of high image diversity, a SME has more chances to build unique image through cooperating with others and vice versa.
The hypothesis 2 is constructed in order to examine the impact of image diversity on creating differentiation feature through alliance. In Figure 6.12, the horizontal axis denotes different goal settings and the vertical axis indicates the uniqueness score which is between 0 and 1 as well. The higher score means the new image configuration of a new alliance is quite different from others. The formula of the uniqueness score and its rationales are already offered in chapter 4, uniqueness analysis. The process of this experiment is very similar to previous one. It began by randomly selecting businesses as the target businesses and then generated different partner suggestions separately according to those 14 goals settings. The uniqueness
‧
scores are computed and averaged in the end for each goal setting.
Figure 6.12 The level of uniqueness under different level of image diversity setting
The Figure 6.12 and 6.13 demonstrate that the image diversity would influence the businesses to create unique feature. The difference between Figure 6.12 and 6.13 is that they are presented in different perspectives. The former one presents the original uniqueness score and the latter exhibits the level of improvement on uniqueness reflected in higher uniqueness score. As observed in Figure 6.12, the original uniqueness scores in higher image diversity context are greater than the scores in lower image diversity context. However, the level of improvement on uniqueness in higher image context is lower than in low image context. Obviously, it is more difficult to improve the level of uniqueness for business in high level context.
A partial explanation for these results may lie in the fact that the businesses in a high image diversity context are already unique enough. To address that, please see
‧
business is already very unique.The other more likely explanation rests in the nature of the effect of partnerships.
If the business image model is really unique, which means this business has higher intensity of special image elements that others don’t. Consider one of our assumptions of this study is that business can acquire mixing image elements for goal achievement through partnerships. If the missing image elements that are brought in are the common image elements, then it is likely to decrease the level of uniqueness. Such effect is relatively evident when the original business is quite unique. In sum, the hypothesis 2 (In the context of high image diversity, a SME has more chances to build unique image through cooperating with others and vice versa) is not supported.
Figure 6.13 The level of uniqueness improvement under different level of image diversity setting
The level of uniqueness imporvement
Desintation (lower image diversity)
Desintation (higher image diversity)
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
89
different image diversity context
the interval of uniqueness score lower image diversity higher image diversity
0~0.25 20% 18%
0.25~0.5 60% 0%
0.5~0.75 20% 18%
0.75~1 0% 64%
However, there is one thing worth noting that the image diversity is not the only factor influencing the level of uniqueness. Sometimes the goal setting matters, especially when the goal is too common. That is, the images we are trying to create are already to be seen everywhere. Then, it would become harder to build unique feature through the common goal.
6.3.2.3 The level of attractiveness
Hypothesis 3: In the context of high image diversity, a SME has more chances to build attractive image through cooperating with others and vice versa.
The hypothesis 3 was designed to examine the impact of image diversity on building attractive images. Similar to the previous experiment, the ways to perform this experiment have nothing different. As showed in Figure 6.14, all of the level of improvement on attractiveness scores in both contexts are greater than 0. This means businesses are able to improve their attractiveness level through partnerships. The detailed information regarding the meaning of attractiveness score and its rationale are available in chapter 4, attractiveness analysis.
The results also suggested that the higher level of image diversity might have potential impact on improving attractiveness. In the Figure 6.14, the solid line indicates the level of improvement on attractiveness in a lower diversity context and
‧ 國
立 政 治 大 學
‧
N a tio na
l C h engchi U ni ve rs it y
90
the dotted line refers to the level of improvement on attractiveness in a higher diversity context. The solid line almost lies above the dotted line. One possible implication is that businesses have more chances to become more attractive by cooperating with others in higher image diversity context. Therefore, the hypothesis 3
the dotted line refers to the level of improvement on attractiveness in a higher diversity context. The solid line almost lies above the dotted line. One possible implication is that businesses have more chances to become more attractive by cooperating with others in higher image diversity context. Therefore, the hypothesis 3