In this paper, we demonstrated a pruning mechanism design and its integration into ECGA. It may also serve as a basis for developing other techniques for more efficient and robust optimization. Some possible extensions of this work are outlined as follows.
First of all, the immediate direction is to design pruning mechanisms for other EDAs. As illustrated in Section 7.3, we can extend the pruning metric described in this paper to handle network-based models with a Bayesian information criterion.
However, a pruning mechanism for network-based models requires more than that. We also need to consider the possible disruption of variable dependencies after pruning a particular variable. The simplest solution is to consider only those variables that are not depended upon by other variables as possible candidates for pruning. However, the validity of such an approach requires further investigation. A more promising yet more sophisticated approach is to first identify the tightly related components (e.g. cliques or strongly connected subgraphs) in the model and then process each component as a unit which is similar to how we process the marginal product models in this work.
Another direction for future research is to assist efficiency enhancement techniques that use the information contained in the built model. As described previously in Section 8.1, some model-based efficiency enhancement techniques for EDAs crucially rely on the structural accuracy of the probabilistic models. However, most of those studies implicitly assume the information contained in the given population is suffi-cient for learning accurate model structures. As demonstrated in the previous sections by nonuniformly scaled problems, this assumption does not always hold. From this perspective, incorporating pruning mechanisms to preprocess the built model for these enhancement techniques is a promising direction for designing more robust approaches.
From an abstract point of view, this work also demonstrates an instance of a new class of techniques operating on built models to control, adapt, or regulate the opti-mization process. Another example based on this viewpoint is the termination criterion proposed by Ocenasek (2006) which uses an entropy-based measurement to evaluate the built model for detecting an appropriate stopping point. According to the informa-tion collected in the model, we can gain better control over the process compared to the conventional evolutionary algorithms. Such an idea may be carried over to other designs of EDAs so that more robust and efficient optimization can be realized.
Acknowledgments
The work was supported in part by the National Science Council of Taiwan under Grant NSC-98-2221-E-009-072. The authors are grateful to the National Center for High-performance Computing for computer time and facilities.
References
Baluja, S. (1994). Population-based incremental learning: A method for integrating genetic search based function optimization and competitive learning. Tech. Rep. CMU-CS-94-163, Carnegie Mellon University, Pittsburgh, Pennsylvania.
Baluja, S., and Davies, S. (1997). Using optimal dependency-trees for combinational optimization.
In Proceedings of the 14th International Conference on Machine Learning (ICML ’97), pp. 30–38.
Bosman, P. A. N., and Thierens, D. (1999). Linkage information processing in distribution estima-tion algorithms. In W. Banzhaf, J. Daida, A. E. Eiben, M. H. Garzon, V. Honavar, M. Jakiela, and R. E. Smith (Eds.), Proceedings of the Genetic and Evolutionary Computation Conference 1999 (GECCO-99), Vol. 1, pp. 60–67.
Chen, Y.-p., and Goldberg, D. E. (2005). Convergence time for the linkage learning genetic algorithm. Evolutionary Computation, 13(3):279–302.
Cover, T. M., and Thomas, J. A. (1991). Elements of information theory. New York: Wiley-Interscience.
De Bonet, J., Isbell, C., and Viola, P. (1997). MIMIC: Finding optima by estimating probability densities. In M. C. Mozer, M. I. Jordan, and T. Petsche (Eds.), Advances in Neural Information Processing Systems, Vol. 9, pp. 424–430.
Deb, K., and Goldberg, D. E. (1993). Analyzing deception in trap functions. In Foundations of Genetic Algorithms 2, pp. 93–108.
Deb, K., and Goldberg, D. E. (1994). Sufficient conditions for deceptive and easy binary functions.
Annals of Mathematics and Artificial Intelligence, 10(4):385–408.
Echegoyen, C., Lozano, J. A., Santana, R., and Larra ˜naga, P. (2007). Exact Bayesian network learning in estimation of distribution algorithms. In Proceedings of the 2007 IEEE Congress on Evolutionary Computation (CEC 2007), pp. 1051–1058.
Etxeberria, R., and Larra ˜naga, P. (1999). Global optimization using Bayesian networks. In A. O.
Rodriguez, M. S. Ortiz, and R. S. Hermida (Eds.), Proceedings of the Second Symposium on Artificial Intelligence (CIMAF-99), pp. 332–339.
Goldberg, D. E. (1989). Genetic algorithms in search, optimization and machine learning. Reading, MA: Addison-Wesley.
Goldberg, D. E. (2002). The design of innovation: Lessons from and for competent genetic algorithms.
Dordrecht, The Netherlands: Kluwer Academic Publishers.
Goldberg, D. E., Deb, K., and Clark, J. H. (1992). Genetic algorithms, noise, and the sizing of populations. Complex Systems, 6(4):333–362.
Goldberg, D. E., Deb, K., and Korb, B. (1990). Messy genetic algorithms revisited: Studies in mixed size and scale. Complex Systems, 4(4):415–444.
Goldberg, D. E., and Rudnick, M. (1991). Genetic algorithms and the variance of fitness. Complex Systems, 5(3):265–278.
Gr ¨unwald, P. (2007). The minimum description length principle. Cambridge, MA: MIT Press.
Harik, G. R. (1997). Learning gene linkage to efficiently solve problems of bounded difficulty using genetic algorithms. PhD thesis, University of Michigan, Ann Arbor.
Harik, G. R. (1999). Linkage learning via probabilistic modeling in the ECGA. IlliGAL Re-port No. 99010, Illinois Genetic Algorithms Laboratory, University of Illinois at Urbana-Champaign.
Harik, G. R., Lobo, F. G., and Goldberg, D. E. (1999). The compact genetic algorithm. IEEE Transactions on Evolutionary Computation, 3(4):287–297.
Hauschild, M., Pelikan, M., Lima, C. F., and Sastry, K. (2007). Analyzing proba-bilistic models in hierarchical BOA on traps and spin glasses. In Proceedings of ACM SIGEVO Genetic and Evolutionary Computation Conference (GECCO-2007), pp. 523–
530.
Hauschild, M. W., Pelikan, M., Sastry, K., and Goldberg, D. E. (2008). Using previous models to bias structural learning in the hierarchical BOA. In Proceedings of ACM SIGEVO Genetic and Evolutionary Computation Conference (GECCO-2008), pp. 415–422.
Holland, J. H. (1992). Adaptation in natural and artificial systems. Cambridge, MA: MIT Press.
Larra ˜naga, P., and Lozano, J. A. (2001). Estimation of distribution algorithms: A new tool for evolution-ary computation, Vol. 2 of Genetic Algorithms and Evolutionevolution-ary Computation. Dordrecht, The Netherlands: Kluwer Academic Publishers.
Lima, C. F., and Lobo, F. G. (2004). Parameter-less optimization with the extended compact genetic algorithm and iterated local search. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2004), pp. 1328–1339.
Lima, C. F., Lobo, F. G., and Pelikan, M. (2008). From mating pool distributions to model overfit-ting. In Proceedings of ACM SIGEVO Genetic and Evolutionary Computation Conference (GECCO-2008), pp. 431–438.
Lima, C. F., Pelikan, M., Goldberg, D. E., Lobo, F. G., Sastry, K., and Hauschild, M. (2007). Influence of selection and replacement strategies on linkage learning in BOA. In Proceedings of 2007 IEEE Congress on Evolutionary Computation (CEC 2007), pp. 1083–1090.
Lima, C. F., Pelikan, M., Sastry, K., Butz, M. V., Goldberg, D. E., and Lobo, F. G. (2006). Substructural neighborhoods for local search in the Bayesian optimization algorithm. In Proceedings of the 9th International Conference on Parallel Problem Solving from Nature (PPSN IX), pp. 232–
241.
Lima, C. F., Sastry, K., Goldberg, D. E., and Lobo, F. G. (2005). Combining competent crossover and mutation operators: A probabilistic model building approach. In Proceedings of ACM SIGEVO Genetic and Evolutionary Computation Conference (GECCO-2005), pp. 735–742.
Lobo, F. G., Goldberg, D. E., and Pelikan, M. (2000). Time complexity of genetic algorithms on exponentially scaled problems. In D. Whitley, D. Goldberg, E. Cant ´u-Paz, L. Spector, I. Parmee, and H.-G. Beyer (Eds.), Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2000), pp. 151–158.
Mitchell, T. M. (1997). Machine learning. New York: McGraw-Hill Higher Education.
M ¨uhlenbein, H., and H ¨ons, R. (2005). The estimation of distributions and the minimum relative entropy principle. Evolutionary Computation, 13(1):1–27.
M ¨uhlenbein, H., and Mahnig, T. (1999). FDA: A scalable evolutionary algorithm for the optimiza-tion of additively decomposed funcoptimiza-tions. Evoluoptimiza-tionary Computaoptimiza-tion, 7(4):353–376.
M ¨uhlenbein, H., and Paaß, G. (1996). From recombination of genes to the estimation of distribu-tions. I. Binary parameters. In Proceedings of the 4th International Conference on Parallel Problem Solving from Nature (PPSN IV), pp. 178–187.
Ocenasek, J. (2006). Entropy-based convergence measurement in discrete estimation of distribu-tion algorithms. In J. A. Lozano, P. Larra ˜naga, I. Inza, and E. Bengoetxea (Eds.), Towards a new evolutionary computation. Advances in estimation of distribution algorithms, Vol. 192 of Studies in Fuzziness and Soft Computing (pp. 39–50). Berlin: Springer.
Pelikan, M., Goldberg, D. E., and Cant ´u-Paz, E. (1999). BOA: The Bayesian optimization algorithm.
In W. Banzhaf, J. Daida, A. E. Eiben, M. H. Garzon, V. Honavar, M. Jakiela, and R. E. Smith (Eds.), Proceedings of the Genetic and Evolutionary Computation Conference GECCO-99, pp. 525–
532.
Pelikan, M., Goldberg, D. E., and Lobo, F. G. (2002). A survey of optimization by building and using probabilistic models. Computational Optimization and Applications, 21(1):5–20.
Pelikan, M., Goldberg, D. E., and Sastry, K. (2001). Bayesian optimization algorithm, decision graphs, and Occam’s razor. In L. Spector, E. D. Goodman, A. Wu, W. B. Langdon, H.-M.
Voigt, M. Gen, S. Sen, M. Dorigo, S. Pezeshk, M. H. Garzon, and E. Burke (Eds.), Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2001), pp. 519–526.
Pelikan, M., and Lin, T.-K. (2004). Parameter-less hierarchical BOA. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2004), pp. 24–35.
Pelikan, M., and M ¨uhlenbein, H. (1999). The bivariate marginal distribution algorithm. In R. Roy, T. Furuhashi, and P. K. Chawdhry (Eds.), Advances in soft computing—Engineering design and manufacturing (pp. 521–535). Berlin: Springer.
Pelikan, M., Sastry, K., and Goldberg, D. E. (2002). Scalability of the Bayesian optimization algorithm. International Journal of Approximate Reasoning, 31(3):221–258.
Rissanen, J. (1978). Modelling by shortest data description. Automatica, 14:465–471.
Sastry, K. (2001). Evaluation-relaxation schemes for genetic and evolutionary algorithms. Master’s thesis, University of Illinois at Urbana-Champaign. Also IlliGAL Report No. 2002004.
Sastry, K., Abbass, H. A., Goldberg, D. E., and Johnson, D. D. (2005). Sub-structural niching in estimation of distribution algorithms. In Proceedings of ACM SIGEVO Genetic and Evolutionary Computation Conference (GECCO-2005), pp. 671–678.
Sastry, K., and Goldberg, D. E. (2004). Designing competent mutation operators via probabilistic model building of neighborhoods. In Proceedings of the Genetic and Evolutionary Computation Conference (GECCO-2004), pp. 114–125.
Sastry, K., Lima, C. F., and Goldberg, D. E. (2006). Evaluation relaxation using substructural information and linear estimation. In Proceedings of ACM SIGEVO Genetic and Evolutionary Computation Conference (GECCO-2006), pp. 419–426.
Sastry, K., Pelikan, M., and Goldberg, D. E. (2004). Efficiency enhancement of genetic algo-rithms via building-block-wise fitness estimation. In Proceedings of the 2004 IEEE Congress on Evolutionary Computation (CEC 2004), pp. 720–727.
Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2):461–464.
Thierens, D., Goldberg, D. E., and Pereira, ˆA. G. (1998). Domino convergence, drift and the tem-poral salience structure of problems. In Proceedings of the 1998 IEEE International Conference on Evolutionary Computation, pp. 535–540.
Yu, T.-L., Sastry, K., Goldberg, D. E., and Pelikan, M. (2007). Population sizing for entropy-based model building in discrete estimation of distribution algorithms. In Proceedings of ACM SIGEVO Genetic and Evolutionary Computation Conference (GECCO-2007), pp. 601–608.