Performance evaluation for Case I

A case study with four different service consumers and ten different airlines was adopted to evaluate the comparative performance of three different approaches. The proposed approach Moderated Fuzzy Discovery Method (MFDM) is evaluated in comparison to the Capability Discovery Method (CDM) and the Fuzzy Discovery Method (FDM) [17].

5.1.2.1 Capability Discovery Method (CDM)

In the first experiment, service discovery approach is based on the use of UDDI registry and the capability search mechanism without involving any fuzzy discovery and higher level abstraction mechanisms (quality rating). This is called the Capability Discovery Method.

The capability matchmaker suggests all the ten Web services to the consumers, since they satisfy the capability constraints (flight booking service). Thus, each Web service consumer starts to check whether the actual contents of the Web services can meet their requirements or not. Figure 5-2 illustrates the fuzzy sets for service consumers that appear in this case and Table 5-1 shows the results related to the precision rate.

In Table 5-1, service Consumer 1’sfuzzy setforCheap is ~( , , , )

1 1 1 1

1 a b c d

C =(0, 0, 13500,

16500). It means that Consumer 1 has a subjective opinion on cheap flight price which is between 0 and 16500. As a result, there are only seven airline Web services can meet Consumer 1’srequirement. So the precision rate is 70% (7 / 10 = 0.7). Use the same principle and apply it to service Consumer 2 , 3 and 4, then different precision rates can be obtain at 0.5, 0.7, and 0.1 respectively.

Table 5-1 CDM precision rates for service Consumer 1 to 4 CDM Suggestions

(No filtering) C1 C2 C3 C4

ChinaEasternAir ˇ ˇ

DragonAir

FarEasternAir ˇ ˇ ˇ ˇ

MacauAir ˇ ˇ

TransasiaAir ˇ ˇ ˇ

JapanAsiaAir

ChinaAir ˇ ˇ ˇ

CathayAir

EvaAir ˇ ˇ ˇ

ShanghaiAir ˇ ˇ ˇ

Precision Rate 7 / 10 = 0.7 5 / 10 = 0.5 7 / 10 = 0.7 1 / 10 = 0.1

5.1.2.2 Fuzzy Discovery Method (FDM)

The second set of experiments in this case is carried out to test the Fuzzy Discovery Method (FDM) [17]. FDM was deployed after the fuzzy classification had been conducted on the underlying data about each service. In this experiment, Fuzzy Classifier adopts the arbitrary C~_init (0,0,14500,16500), where abcd (see Figure 5-1), as the fuzzy rule for classification according to the actual cost of a specific flight. Before the FDM is applied for service discovery, each of the ten services will be rated by C~_init

and therefore each service gets a value representing its higher level informative declaration (quality rating or QoS) on the primitive term Cheap.

Before the FDM can be deployed, the Fuzzy Classifier have to conduct fuzzy classification on the data provided by each service provider. The initial fuzzy set,

) 16500 , 14500 , 0 , 0

~_init (

C , is introduced to calculate primitive term Cheap for each service provider. The classification results are shown in Table 5-2.

Table 5-2 Classification results for each service with C~_init (0,0,14500,16500) Service QoS Value

for Cheap Service QoS Value

for Cheap

ChinaEasternAir 0.4 JapanAsiaAir 0

DragonAir 0 ChinaAir 0.16

FarEasternAir 0.5 CathayAir 0

MacauAir 0.47 EvaAir 0.23

TransasiaAir 0.52 ShanghaiAir 0.16

Suppose that the threshold θ = 0.25 isadopted forallweb consumers. θ,thethreshold, is used in the Fuzzy Discovery to filter out those services that are less likely to meet the requirement. In this experiment, Fuzzy Discovery only recommends four possible satisfactory Web services, that is, ChinaEasternAir, FareasternAir, MacauAir and TransasiaAir.

Consumer 2 with fuzzy set ~ ( , , , )

2 2 2 2

2 a b c d

C =(0, 0, 14500, 14500) indicates that his / her subjective cheap price sits between 0 and 14500. From the evaluation result shown in Table 5-3, it can be observed that only two flight booking services can satisfy his / her requirement.

For service Consumer 2, the precision rate is 50% (2 / 4 = 0.5). In addition, the same principle can be also applied to Consumer 1, 3, and 4 and the results are 100%, 100%, and 25% respectively for the precision rates.

Table 5-3 FDM precision rates for Consumer 1 to 4 with θ = 0.25 θ = 0.25

FDM Suggestions C1 C2 C3 C4

ChinaEasternAir ˇ ˇ

FarEasternAir ˇ ˇ ˇ ˇ

MacauAir ˇ ˇ

TransasiaAir ˇ ˇ ˇ

Precision Rate for Specific Consumer 4 / 4 = 1 2 / 4 = 0.5 4 / 4 = 1 1 / 4 = 0.25

Ifθ is0.5,only FareasternAirand TransasiaAirwillberecommended and theprecision rates for FDM are revealed in Table 5-4.

Table 5-4 FDM precision rates for Consumer 1 to 4 with θ = 0.5 θ= 0.5

FDM Suggestions C1 C2 C3 C4

FarEasternAir ˇ ˇ ˇ ˇ

TransasiaAir ˇ ˇ ˇ

Precision Rate for Specific Consumer 2 / 2 = 1 2 / 2 = 1 2 / 2 = 1 1 / 2 = 0.5

5.1.2.3 Moderated Fuzzy Discovery Method (MFDM)

The third set of experiments is conducted to test the Moderated Fuzzy Discovery Method (MFDM). After four service consumers have made the requests via the Fuzzy Discovery and give their feedbacks or opinions on the primitive term Cheap. The SAM method will be conducted by Fuzzy Moderator to aggregate the group consensus on primitive term Cheap in order to produce a more objective inference rule. This process has been detailed in section 5.1.1 and a moderated consensus value for primitive term Cheap is derived as

14925.007) 13314.333,

0,

~ (0,

C . This consensual value will replace the existing one

(C~_init (0,0,14500,16500)). With the new derived fuzzy set, Fuzzy Classifier will be triggered again in order to obtain new classification result for the term Cheap. This is illustrated in Table 5-5.

Table 5-5 Classification results for each service with moderated C~ (0,0,13314.333,14925.007) Service QoS Value

for Cheap Service QoS Value

for Cheap

ChinaEasternAir 0.01 JapanAsiaAir 0

DragonAir 0 ChinaAir 0.14

FarEasternAir 0.5 CathayAir 0

MacauAir 0.1 EvaAir 0.08

TransasiaAir 0.26 ShanghaiAir 0.13

In this experiment, only two flight booking services are above the threshold θ = 0.25, that is, only two possible Web service, FarEasternAir and TansasiaAir, will be recommended

by Fuzzy Discovery. Consumer 3 with fuzzy set ~ ( , , , )

3 3 3 3

3 a b c d

C =(0, 0, 14000, 15500)

indicates that his / her subjective cheap price sits between 0 and 15500. From the result shown in Table 5-6, two of the recommended flight booking services can satisfy service Consumer 3’ssubjective opinion. The precision rate has increased to 100% (2 / 2 = 1), due to the contribution of the proposed moderation. By applying the same steps to the other service Consumers 1, 2, and 4, their precision rates would therefore be 100%, 100%, and 50%

respectively.

Table 5-6 MFDM precision rates for Consumer 1 to 4 with θ = 0.25 θ= 0.25

MFDM Suggestions C1 C2 C3 C4

FarEasternAir ˇ ˇ ˇ ˇ

TransasiaAir ˇ ˇ ˇ

Precision Rate for Specific Consumer 2 / 2 = 1 2 / 2 = 1 2 / 2 = 1 1 / 2 = 0.5

Ifθ is0.5,only FarEasternAirwillberecommended and theprecision rates for MFDM are revealed Table 5-7.

Table 5-7 MFDM precision rates for Consumer 1 to 4 with θ = 0.5 θ= 0.5

MFDM Suggestions C1 C2 C3 C4

FarEasternAir ˇ ˇ ˇ ˇ

Precision Rate for Specific Consumer 1 / 1 = 1 1 / 1 = 1 1 / 1 = 1 1 / 1 = 1

5.1.2.4 Summary of Case I

Table 5-8 shows an integrated view of Table 5-1,Table 5-3,Table 5-4,Table 5-6 and Table 5-7. The average precision rates for CDM, FDM and MFDM are indicated in Table 5-8 with different thresholds.

From Table 5-8, it can be concluded that the proposed Moderated Fuzzy Discovery Method (MFDM) has outperformed the Fuzzy Discovery Method (FDM) and the FDM has

produced better precision rate than the Capability Discovery Method (CDM). In addition, MFDM has performed twice as well as the CDM in terms of precision rate.

Table 5-8 Precision rates for CDM, FDM and MFDM with different thresholds Precision Rates for

Specific Consumer C1 C2 C3 C4 Average

Precision Rate

θ 0.25 0.5 0.25 0.5 0.25 0.5 0.25 0.5 0.25 0.5

CDM 0.7 0.5 0.7 0.1 0.5

FDM 1 1 0.5 1 1 1 0.25 0.5 0.68 0.87

MFDM 1 1 1 1 1 1 0.5 1 0.87 1

Through the consideration of quality rating on the perspective Cheap, and the use of the proposed moderation process, the precision rate of service discovery can be improved by pre-classifying services and filtering out those services whose quality of underlying content is not considered as a recommended service. This will save the consumers’time while selecting the suitable services.

The results show that CDM is the most imprecise way for service discovery.

Nevertheless, CDM uses general UDDI inquiries where no additional pre-classification is needed before service discovery. Both of FDM and MFDM need the additional computation cost for classification (time for evaluating the QoS terms of all services). In this experiment, the time for pre-classification process is less than 1 second. MFDM consumes extra 0.921875 second for SAM processing time. Briefly, if CDM is treated as a basis, then FDM consumes less than 1 additional second and MFDM requires an extra 1.732875(+-0.5) seconds. The additional time is trivial but it does increase the computational cost when FDM and MFDM are applied. The cost might vary according to the amount of data and the number of feedback classifications. Considering the time gained from the increase of precision rate and the time saved by filtering out the less significant services, MFDM is a better solution for service discovery.

In Case I, however, only one perspective, Cheap, is used. Different weightings from different service consumers for the ingredients of a composite term are not considered. In addition, the number of consumers is small in Case I. Therefore, in the next case study, a larger scale of exercises with multiple criteria will be conducted in order to examine the issues associated with scalability.