According to a recent survey of 24 existing service composition frameworks in perva-sive environments [33], 17 of them are categorized as Type-based Service Composition (TBSC), since they only compose the service by simply matching node types; the re-maining 7 of them match the values of attributes against a set of user-specified expres-sion, which are called Expression-driven Service Composition (EDSC). In the following, the negotiable and unifiable service composition approach is called the Negotiable-Expression-driven Service Composition (NESC). This work evaluates the quality of the above mentioned approaches based on several metrics. The detailed discussions and experimental results of these metrics are reported below.
All experiments are conducted on P4 1GHz CPU PCs with 1GB memory and all input data are randomly generated to simulate the real world situation. In each experiment, the number of Service Request is set to 1000, the lengths of services are randomly distributed from 3 to 5, and there are totally 15 node types in the system.
Each composition method is performed to select candidates among a group of Worker Nodes ranging from 500 to 1500 instances and each node consists of 7 to 11 attributes.
Among these attributes, 50% of them are constrained by user preferences. By default, the number of mandatory preferences is equal to the number of negotiable preferences.
Success Rate of Composition (SRC)
One simple way to evaluate the quality of a service composition mechanism is to cal-culate the Success Rate of Composition (SRC) [80]. SRC is defined as follows:
SRC = n(Ssuccess)
n(S) , (5.24)
where Ssuccess is the set of services that are successfully composed, and S is the set of services to be composed. Recall that n(S) is the cardinality of the set S (see Definition 7).
As depicted in Figure 5.22, along with increasing of the number of nodes, the success rate for EDSC and NESC (the proposed approach) are also slightly increased, whereas TBSC is steady at approximately 75%. TBSC always has the best SRC score since its selection criteria are less restrictive. More specifically, only the type information is used as a constraint. On the other hand, EDSC has poor SRC, since its Node Preference Descriptor is more restrictive. It is worthy to point out that the SRC of the NESC is higher than that of EDSC because the inclusion of negotiation capability.
The major issue of SRC metric is that it is a coarse-grained measure of the success rate of service composition. Specifically, a service is successfully composed if and only if all of the desired Worker Nodes are found. Assuming that there are n desired Worker Nodes, then the composition fails even when n− 1 out of n valid nodes are found.
Hence, SRC largely depends on desired Worker Nodes specified in the Service Request so that it can be inaccurate to evaluate the quality of a service composition mechanism by using only the SRC metric.
Success Rate of Matching (SRM)
To deal with the issue mentioned above, a finer-grained metric called the Success Rate of Matching (SRM) is proposed below:
500 1000 1500
Figure 5.22: Success Rate of Composition (SRC)
SRM =
where Wsvalid is set of Worker Nodes that are successfully found and matched for the service s, S is the set of services to be composed, and ℓ(s) is the length of s which is defined in the Definition (8).
The core idea of SRM is to measure the success rate based on the number of successfully found nodes instead of the number of successfully composed services. When a service needs to be composed and if there are n− 1 out of n nodes which are found, then n−1n is given to SRM instead of 0. Figure 5.23 shows SRM with different number of Worker Nodes. Both the SRMs of EDSC and NESC slightly increase when the number of nodes is increased. Again,TBSC has the highest SRM. It is interesting to point out that all evaluated approaches have higher score in SRM than in SRC, since SRC is an
”all or nothing” type metric. Also note that the SRM scores of both EDSC and NESC
500 1000 1500
Figure 5.23: Success Rate of Matching (SRM) with different number of Worker Nodes are higher than that of SRC.
A composition mechanism with high SRM does not guarantee high quality of ser-vice. In an extreme case, a composition mechanism can achieve high SRM by simply reporting that all nodes are candidates. As a result, a metric that measures the preci-sion of the timing for reporting candidates is required.
Recently, many researchers found that ”recall” and ”precision”, which are common evaluation measures in the information retrieval field [98], are very useful metrics for evaluating the quality of service composition [121, 117]. According to (5.23), NESC only returns the best node so that n(Wsrequested) represents the total number of relevant nodes of a Service Request. Therefore, the semantics of SRM is identical to the concept of ”recall” which is the number of relevant items retrieved (i.e. ∑
s∈S
n(Wsvalid)) over the number of total number of relevant items (i.e. ∑
s∈S
n(Wsrequested)). In the following, the metrics for measuring the precision of a composed service will be presented.
Precision of Composition (PoC)
Precision is the number of relevant items retrieved over the number of total retrieved items [98]. Hence, the Precision of Composition (PoC) can be defined as the number of valid nodes retrieved over the number of total retrieved nodes, namely,
P oC =
where n(Wsvalid) is the number of nodes that fulfills the corresponding Node Preference Descriptors and n(Wsf ound) is the number of nodes found by the PSM.
Figure 5.24 illustrates the PoC of the three approaches with increasing number of nodes. The PoCs of TBSC, EDSC, and NESC are steady at 10%, 78%, and 86%, respectively. It is important to note that the although TBSC gets high SRC/SRM in the previous experiments, it suffers from extremely low PoC. In other words, TBSC tends to retrieve too many candidates causing the precision being extremely low. On the contrary, EDSC is too restrictive so that, although it gets the highest PoC, the success rate (SRC/SRM) is poor. Figure 5.23 and 5.24 show that NESC is able to maintain high score both in SRM/SRC and PoC. Specifically, the NESC is precise enough so that it is able to compose high quality services while maintaining reasonable success rate of composition.
Also, from Fig. 5.24, given that the number of constraints on node attributes are the same, PoC is independent of the number of the Worker Nodes. Therefore, additional experiments are performed to observe the relationship between PoC and the ratios of constrained attributes. The outcomes are shown in Figure 5.25. Observe that the PoCs of TBSC drop rapidly whereas PoCs of other approaches increase gradually.
The results show that TBSC is more inappropriate if there are more constraints on attributes.
500 1000 1500
Figure 5.24: Precision of Composition (PoC) with different number of Worker Nodes
20 30 40 50 60 70
Figure 5.25: Precision of Composition (PoC) with different ratio of constrained at-tributes