5 Performance Evaluation
5.2 Experimental Results
5.2.1 The Evolution of Scheme GA
As mentioned in Section 4, scheme GA is an iterative process. The quality of obtained result and execution time is dependent on the value of nGen. Obviously, the obtained result is of better quality when nGen is larger. However, a large nGen implies long execution time. Hence, it is important to determine a proper value of nGen to strike a balance between the execution time and the quality of result. In this experiment, we investigate the evolution process of scheme GA by varying the value of nGen. Figure 9 shows the effect of different values of nGen ranging from 0 to 500 on the average access times of the resulting broadcast programs. The corresponding execution times of all schemes are presented in Figure 10. Note that nGen= 0 represents the case of randomly generating nP op solutions.
As shown in Figure 9, since scheme OPT, GREEDY and VFK are independent of nGen, the average access times of the result broadcast programs of these schemes are not affected by nGen. We observe
0
Figure 9: The average access time with nGen varied
Figure 10: The execution time with nGen var-ied
that the results of VFK are much worse than that of other schemes, and hence, the results of VFK are omitted henceforth. Although the results of VFKare close to the optimal ones when the data items are independent of one another, scheme VFK does not perform well due to the lack of the consideration of data dependency. This result shows the necessity of the consideration of data dependency. We also observe that scheme GREEDY outperforms scheme VFK. This can be explained by the reason that scheme GREEDY considers the data dependency by placing the required data items of the queries with higher access probabilities as closely as possible.
The average access times of the broadcast programs of scheme GA decrease as the value of nGen increases. In addition, the results obtained by scheme GA become closer to the optimal ones when the value of nGenis larger. However, the speed of convergence becomes slow when nGenis larger than 200.
Therefore, we set nGento be 200 in the following experiments. We observe that when nGen is small, scheme GREEDY outperforms scheme GA. However, when nGenis large enough (larger than 80 in this experiment), the results of scheme GA are better than that of scheme GREEDY. The performance gain of scheme GA over GREEDY increases from 17% to 31.4% when nGenincreases from 100 to 500.
0
Figure 11: The average access time with n var-ied
Figure 12: The execution time with n varied
As shown in Figure 10, the execution time of scheme GA increases linearly as the value of nGen increases, which fully agrees with the time complexity analysis in Section 4.2. Since scheme VFK and GREEDY are simple heuristics, they execute much faster than scheme GA. In addition, since the execution time of scheme OPT is much longer than that of other schemes, the execution time of scheme OPT is omitted in this and the following experiments.
5.2.2 The Effect of the Number of Channels
This experiment investigates the effect of the number of channels in the average access times and the execution times of all schemes. Figures 11 and 12 show the average access times and execution times of scheme GA, GREEDY and OPT with the number of broadcast channels (i.e., n) varied. The value of n ranges from 2 to 10.
Consider the results shown in Figure 11. The average access times of all schemes decrease as the number of channels increases. This result agrees with our intuition in that the increase of the network bandwidth causes the average access time to decrease. However, the improvement on the average access time decreases as the number of channels increases. As a result, the determination of the number of
broadcast channels should consider the balance between performance improvement and the number of channels used. The number of broadcast channels suggested by our experiment is around 6. The performance gain of scheme GA over scheme GREEDY increases from 13.33% to 25.67% when the number of channels increases from 2 to 10. This result shows that scheme GA outperforms scheme GREEDY especially when the number of channels is large.
As shown in Figure 12, the execution time of scheme GREEDY is not affected by the number of channels. However, the execution time of scheme GA decreases as the number of channels increases.
This result is caused by query refinement. By Definition 3, a large value of n implies a short broadcast program (i.e., a small value of L). According to function QueryRefinement, the number of the required data items of each refined query Q0i is always smaller than or equal to L. The decrease of L will incur short refined queries and therefore decrease the time to calculate TStartup(Q0i) and TRetr.(Q0i).
5.2.3 Comparison of Single and Multiple Channel Environments
We next investigate the effect of the number of channels by fixing the total available bandwidth. We set the total bandwidth to be 800K bytes/sec, and the number of channels is set to be from 1 to 10. The average access times of all schemes with the number of channels varied are given in Figure 13.
As observed, the average access times of all schemes increase as the number of channels increases.
This result agrees with the intuition that in a single environment, all bandwidth can be best utilized to minimize the average access time. The utilization of bandwidth degrades in a multiple channel environment since a mobile client can only listen on one channel at a time. Multiple channels also increase the difficulty to minimize the average access time. Hence, the performance degradation of scheme GA over scheme OPT increases from 0.96% to 8.02% as the number of channels increases
0
Figure 13: The average access time of single and multiple channel(s) with fixed total band-width
Figure 14: The average access time with the value of fanout varied
from 1 to 10. Under the same condition, it is interesting to see that the performance degradation of scheme GREEDY over scheme OPT increases from 5.64% to 48.88%. This is explained by the reason that scheme GREEDY only tries to place data items with the same query as closely as possible and does not consider the effect of multiple channels. As a result, the performance degradation of scheme GREEDY over scheme OPT becomes more severe when the number of channels increases.
5.2.4 The Effect of Average Fanout of Data Items
This experiment evaluates the effect of the average fanout of data items. Figure 14 shows the average access times of all schemes with the value of average fanout varied. The value of average fanout is set to be from 5 to 30. As shown in Figure 14, the average fanout of data items only slightly affects the results of all schemes. The average access time of scheme OPT ranges from 9.8 to 10.2. In addition, scheme GA outperforms scheme GREEDY in this experiment. The performance gain of scheme GA over scheme GREEDY ranges from 16.62% to 26.41%.
0
Figure 15: The average access time with aver-age query length varied
Figure 16: The average access time with θ var-ied
5.2.5 The Effect of Average Query Length
In this experiment, we investigate the effect of the average query length. Figure 15 shows the average access times of all schemes with the value of average query length varied. The value of average query length is set from 5 to 30. It is intuitive that the average access time increases as the value of average query length increases. We observe that the performance degradation of scheme GA over scheme OPT increases from 8.24% to 13.67% when the value of average query length increases from 5 to 15. In addition, the performance degradation of scheme GA over scheme OPT keeps in the range between 11.5% and 13.5% when the average query length is larger than 15. On the other hand, the performance degradation of scheme GREEDY over scheme OPT increases from 15.11% to 41.95%. In addition, the performance degradation of scheme GREEDY over scheme OPT keeps in the range between 34%
and 43.5% when the average query length is larger than 15. It can be explained that the optimization constraints are relaxed when the value of query length is small. Hence, all schemes perform well with a small value of average query length. In addition, scheme GA outperforms scheme GREEDY especially when the value of average query length is large. Since scheme GREEDY is a simple heuristic, the
performance of scheme GREEDY degrades severer than that of scheme GA when the optimization constraints are strict (i.e., the value of average query length is large).
5.2.6 The Effect of the Skewness of Queries
In this experiment, we consider the effect of the skewness of the access probabilities of queries. The skewness is controlled by the value of θ. The larger the value of θ, the more skewed the access proba-bilities of the queries are. The value of θ is set to be from 0 to 1.2. Note that θ = 0 indicates that the access probabilities of all queries are uniform (i.e., P r(Qi) = P r(Qj) for all i and j).
Figure 16 shows the average access times of all schemes with θ varied. We observe that the average access times of all schemes decrease as the value of θ increases. It is because that when the access probabilities are not skewed, minimizing the average access time is not effective since more queries are involved. When the access probabilities of queries are skewed, optimizing hot queries is able to minimize the average access time well. Hence, scheme GREEDY performs better when the access probabilities are highly skewed since scheme GREEDY favors queries with high access probabilities.
As a result, the performance gain of scheme GA over scheme GREEDY decreases from 27.79% to 13.73% as the value of θ increases from 0 to 1.2.
5.2.7 Summary
Based on the above experimental results, the characteristics of scheme GA and GREEDY are summa-rized in this subsection according to the following respects.
• Result effectiveness: The relative performance of schemes depends on the value of nGen. If nGen
is large enough (i.e., given sufficient time to execute), scheme GA is more effective than scheme
GREEDY. However, if the time to execute is not sufficient (i.e., nGen is too small), scheme GREEDY outperforms scheme GA.
• Execution speed: Scheme GREEDY is faster than scheme GA due to the simplicity of scheme GREEDY.
• Performance stability: If nGen is large enough (i.e., given sufficient time to execute), the results of scheme GA are always close to the optimal ones. On the other hand, the performance of the results of scheme GREEDY depends on several factors. For example, as pointed out above, scheme GREEDY does not perform well when the access probabilities of queries are not skewed.
Hence, the performance of scheme GA is more stable than that of scheme GREEDY.
• Convergence: Since genetic algorithms are convergent processes, the results of scheme GA be-come better when the allowed time to execute is longer. On the contrary, the results of scheme GREEDY are not affected by the allowed time to execute.