The Study of Optimal File Allocation in Star Distributed Computing Systems based on Grouping Genetic Algorithm
9
0
0
全文
(2) A survey of reliability design with genetic algorithm based is presented in [6]. Y. S. Yeh, et al [11] for K-node set reliability optimization with capacity constraint of a distributed system. In [13], A. Kumar, et al achieved various files were allocated to different nodes of a distributed computing system so that the reliability of a executing a program was maximized.. 2.3 Mathematical model An optimal file allocation in a star topology problem is described above can be formulated as follows: Maximize: DSR(S). (1). s j ≤ ci ,. (2). Subject to:. In daily life, people usually connect their computer to a hub through communication lines for communicating with others no mater where they are. It is clear to see that most computers are centralized in a hub. In graphic, we can present this connection schema as star topology. We are interested in that how to allocate a set of various files such that the optimal reliability of a star DCS with given the reliability of communication and perfect processing nodes. In this paper, we took advantage of the concept of grouping genetic algorithm to handle it. It would find an exact or near-optimal file allocation in finite execution time.. ∑x. ij. ∑x. ij. j. = fj. (3). i. i = 1,2L, n; j = 1,2, L, m. (4). where DSR(S) is the distributed system reliability of a star network S. xij represents that file j is allocated to node i.. ⎧1 xij = ⎨ ⎩0. 2. Problem statements and Mathematical model. if file j is allocated to node i (5) otherwise.. s j is the size of file j. ci is the capacity of processing node i. f j is the copies of file j. n is the number of processing nodes in S. m is the number of distinct files in the file. Before this file allocation problem proceeded, we firstly introduce the simple alike case, and then give a full description of ours.. set.. 2.1 Bin packing problem. The following list the relevant assumptions of this problem:. The bin packing problem (BPP) is a combinatorial optimization problem and defined as below [15]:. 1.. Given a finite set of the N objects with sizes. 2.. si and the identical bins with capacity C, such that each si ≤ C . Find a partition of the ob-. Both the processing nodes and the central node (hub) in a star network are perfect. Failure of a communication link in a star network is independent of failures of others. At most one copy of each file can be allocated in a processing node.. jects into the bins such that the number of bins, K, is minimized.. 3.. 2.2 File allocation in a star network problem. 2.4 File allocation in a star network problem is NP-hard. File allocation in a star network problem (FASP) can be described as follows: given a set of files of various sizes and copies, subject to physical environment constraints (i.e. reliability and memory of processing nodes, reliability of distributed systems and communication links, cost and so on) , how should be the files distributed among the processing nodes in a star network, so that total size of the files within each processing node are no larger than its capacity and the redundant files do not in the same processing node, as to obtain the best distributed system reliability ?. In this section, we assume that the reader is familiar with the basic notions of NP-hard [15]. The BPP [15] is a well-known NP-hard problem. Therefore, we show that the BPP can be reduced to the FASP, and the FASP is NP-hard as well. The statements of the proof are delineated as below: Proof: Given any instance I of the BPP, which has identical bins of capacity C and a finite set of object of sizes si such that each si ≤ C . Trivially, there is a direct mapping can make the -2-.
(3) called grouping genetic algorithm (GGA). The main steps in GGA are the same as GA. The differences in GGA are that crossover operator transfers groups from parents to offspring and mutation operator works with groups rather than items. The steps in our solution algorithms are described as the following sections.. BPP be reduced to the FASP easily. The direct mapping maps the objects and the bins in the BPP to the files and the processing nodes in the FASP respectively. Here, we assume the mapped star network S in the FASP has the same reliability r (<1) of communication links and each file has only one copy. This S is the simplest special case in the FASP. In addition, we assume the maximal distributed system reliability R of the S can be computed in amount of finite time. That. 3.1 Chromosome encoding We use a binary number based coding scheme. The length of a chromosome, lchrom, depends on n and m, and is equal to m × n , where m is the number of distinct files and n is the number of processing node. An illustration of a chromosome scheme is given in Figure 1, where xij = 1 (0) if file j is (not) allocated to. is, R = r k where the parameter k is the minimal number of the nodes to which the files allocated. The parameter k is able to draw as. log R log r. and also the minimal number of bins in the BPP. Owing to NP-hard problem, there is no known optimal algorithm for BPP running in polynomial time; in other words, the minimal number of bins, k, can’t be computed in amount of finite time. The fact contradicts the assumption. So, the FASP is NP-hard as well ■.. node i. 3.2 Evaluation of chromosome Because computing the reliability of a star DCS has been shown NP-hard [2], it is hard to calculate exact reliability of a DCS. Therefore, we adopt approximate computing methods [2, 3] to be our evaluation of chromosomes. There are two bound for our problem, and they are described respectively as follows:. The FASP has been shown as NP-hard. Thus, as either the number of processing nodes or of the distinct files, or both increase, it would take a large amount of execution time to find an optimal file allocation in a star network.. Procedure : L_BOUND (v) For k = 1 to m do. 3. Solution algorithm based on grouping genetic algorithm. Fk' = Pk ; Ok' = φ ;. In the industrial, many optimal problems are quite hard to solve by conventional optimization techniques. A powerful method originated from natural genetics is called genetic algorithm (GA) which has been developed by John Holland in 1975 can provide a means to handle various optimal problems [16-18]. In GA, a chromosome represents a solution to the problem at hand. Each chromosome is composed of genes. A gene is the basic unit of information. Each gene represents a trait and will influence the future individual. The basic GA operations include crossover, mutation, selection, and fitness evaluation. In [20], E. Falkenauer modified GA for grouping problems. The modified GA is. ← node 1 →. x11. x12. x13. ← node 2 →. x 21. x 22. x 23. / * lower bound * /. For i = 1 to k - 1 do if Pi − Pk = 1 then Ok' = Ok' ∪ (Pi − Pk ) ; End. End. return. m. ∑∏. k =1 j∈Fk'. qj. ∏ pj ;. j∈Ok'. (6). End Procedure.. ← node 3 →. x 31. 1−. x 32. x 33. ← node 4 →. x 41. x 42. x 43. ← node 5 →. x 51. Figure 1 Illustration of a chromosome scheme with n = 5 and m = 3 (15 genes).. -3-. x 52. x 53.
(4) Procedure : U_BOUND(v). / * upper bound * /. fitness (v ) =. For k = 1 to m do Fk" = Pk ; Ok" = T − Pk ;. (. ). if Pi ∩ Ok" − { j} ≠ φ for i = 1, K , k − 1 Ok" = Ok" − { j} ;. End. End. return. 1−. (8). 3.3 Initial population. For j ∈ T − Pk do then. U_BOUND(v) + L_BOUND(v ) 2. m. ∑ ∏q j ∏ p j ;. k =1 j∈ Fk". (7). j∈O k". End Procedure.. In this stage, we use a combinatorial method of random and worst fit (WF) heuristic to produce a chromosome that satisfies the node’s constraints and the file’s constraints. Provided that the selected random node does not contain the file and its capacity is large enough to put, we can put it into the selected random node. If not, require another node. Putting the file into the founded node that with largest capacity and does not yet contain it yields the worst fit heuristic. 3.4 Operators. where. The roulette wheel strategy [16, 17], fitness-proportional selection, is used. Crossover operator transfers some node section from parents to offspring [20]. Mutation operator works with a random node rather than a random bit [20]. Repair operation used to tune those infeasible chromosomes to feasible ones.. Pk is the kth file cutest in the set of file distributed in a star network. T is the set of distinct files distributed in a star network. Ok' , Ok" are the sets of nodes that do not contain file k. Fk' , Fk" are the set of nodes that contain file k. p j is the working probability of link associate with the node j in O k' or O k" . q j is the failure probability of link associate. Owing to the fact that exchanges a section of the chromosome in the crossover stage, the file constraints could be violated. We call the violation of the file constraints as defect phenomenon. It can be divided two case further － positive defects and negative defects. Positive defects refer to the copies of files over than the file constraints; on the contrary, refer as negative defects. In order to tune those infeasible chromosomes to feasible ones, we adopt a pair of repairing mechanisms. The repair procedure is. ' " with the node j in Fk or Fk .. The computational complexity of these two approximate methods are O(mn2) [3]. We take conveniently the mean of the upper and the lower bound as the fitness of a chromosome. That is as follows:. listed as bellows, where RC ' and RC " are the remaining capacity factor for positive and negative defects respectively.. Procedure : REPAIR Calculate the current copies of each file and the current remaining capacity of each node in a chromosome. / * process for positive defects * / For i = 1 to m do times = the current copies of f i - the constraint of f i . if times > 0 then For j = 1 to times do For k = 1 to n do Calculate the RC k' and the selective weight RC k' × q k for node k . End. Select a node which contains f i with largest selective weight and then delete f i and update the capacity of that node. End. End.. -4-.
(5) / * process for negative defects * / For i = 1 to m do times = the constraint of f i - the current copies of f i . if times > 0 then For j = 1 to times do For k = 1 to n do Calculate the RC k" and the selective weight (1 - RC k" ) × p k for node k . End. Select a node with largest selective weight and then add f i and update the capacity of that node. End. End. End Procedure... C++, and listed as below:. '. For the positive defects, the RC of node i refers as follows:. RCi' =. Remaining Capacity of node i n. 1.. , (9). 2.. ∑ Remaining Capacity of node j j =1. 3.. i = 1, 2, K , n For the negative defects, the RC " of node i refers as follows:. 4.. ⎧ Remaining Capacity of node i , ⎪ n ⎪⎪ ∑ Remaining Capacity of node j (10) RC " = ⎨ j =1 ⎪ ⎪ if node i can accommodate the file. otherwise ⎩⎪ 0 ,. weight (i ) =. ci. ( pi ). m ∑ sj j =1. , i = 1, 2, ..., n ,. (11). where ci is the capacity of node i, and sj is the size of file j. 5. Grouping genetic algorithm 1 (GGAFAS 1): without ranking and replacement; that is, only crossover and mutation. 6. Grouping genetic algorithm 2 (GGAFAS 2): ranking [16-18] version. 7. Grouping genetic algorithm 3 (GGAFAS 3): full generational replacement version. 8. Grouping genetic algorithm 4 (GGAFAS 4): elitist [16-18] version.. i = 1, 2, K, n.. 3.5. Exact algorithm (EA) [21, 22]: find the exact reliability with the best file allocation. Heuristic algorithm 1 (HA 1): assign files based on the reliability of a communication link. Heuristic algorithm 2 (HA 2): assign files based on the product of a node capacity and the reliability of the communication link which incident on the node. Heuristic algorithm 3 (HA 3): It’s criterion of the candidate node selection as follows:. Replacement. In order to get better solutions, the worse individuals have to be thrown away in processing. Two replacement used here; one is full generational replacement and the other is elitist strategy [16-18]. The two worst chromosomes are replaced by their offspring soon after they give birth. This is called the full generational replacement.. In the implementation of our heuristics, we also consider the effect affected by the files assigned order. We divided this influence into two ways in detail; one is the smallest file size first and the other is the biggest first. At each pass, files relying on their sizes assigned in turn until satisfy the file requirements.. 3.6 Termination rule We treat the upper bound of generations specified by the tester as the termination condition of the GGAFAS. When the number of generations exceeds the upper bound, the execution of the GGAFAS is terminated.. We used a set of star distributed computing systems with the numbers of various processing nodes, the unlike capacities of processing nodes, the number of distinct files, and the different copies of files as our experimental materials. The first number of the test file name represents that. 4. Experimental results The experimental programs used in our experiments had been implemented in Borland -5-.
(6) the node number in a star DCS, and the second number represents that the distinct file number in a star DCS. For example, a 6-4 test file represents a star DCS consists of 6 different processing nodes and 4 distinct files.. reliability. The data met the exact values shown in Table 1 are underlined, and the most approximate is shown with asterisk (*). When the number of processing nods in a star DCS greater than 20, we calculated the average of upper bound and lower bound as its reliability. We could see that the HA1 performs well than other heuristic algorithms in almost all cases form Table 1.. Due to all of the heuristics have two file assignments, in Table 1, the column title “smallest” refers to the file with smallest size first assigned, and “biggest” refers to the file with biggest size first assigned. Symbol ‘－‘ means that the star DCS is too large to exactly calculate its. Table 1 the experimental data of the exact algorithm and the heuristics. EA. HA 1. smallest. HA 2. biggest. smallest. HA 3. n. m. biggest. smallest. biggest. 6. 4. 0.85344. 0.7576. 0.82754*. 0.721. 0.784. 0.7735. 0.79856. 7. 3. 0.89544. 0.871192. 0.8596. 0.787. 0.8176. 0.872032*. 0.8596. 8. 3. 0.90272. 0.89014*. 0.8806. 0.8575. 0.8398. 0.88804. 0.8785. 9. 3. 0.953907. 0.953907. 0.952828. 0.841704. 0.93761. 0.840854. 0.935636. 10. 3. 0.945002. 0.918492. 0.945002. 0.83814. 0.933372. 0.915672. 0.926386. 15. 6. －. 0.895647. 0.908301. 0.804284. 0.864195. 0.886023. 0.916376*. 20. 13. －. 0.912788. 0.905985. 0.87128. 0.905356. 0.919974*. 0.909575. 30. 19. －. 0.9106. 0.919068. 0.893928. 0.882953. 0.920838*. 0.905563. 50. 22. －. 0.918804*. 0.902886. 0.86129. 0.856423. 0.908406. 0.899141. 100. 40. －. 0.908864*. 0.907111. 0.806761. 0.785517. 0.885185. 0.885948. 200. 25. －. 0.968567*. 0.96558. 0.863665. 0.850215. 0.961191. 0.965395. 2.. In our experiments for GGAFASs, each test file represents a star DCS. According to the number of processing nodes in a DCS, we could classify them into three sorts – small-scale ( n ≤ 10 ) , middle-scale (10 < n ≤ 50 ) ,and large-scale ( n ≥ 50) , where and n is the number of processing node. We chose appropriate population sizes for each sort. Small-scale would have 100 individuals. Middle-scale would have 300 individuals. Large-scale would have 500 individuals. We also found well performance for GGAFASs with crossover rate rc = 1, mutation rate rm = 0.05, and generation=30. The experimental data listed in Table 2.. 3. 4.. We observed both GGAFAS 3 and GGAFAS 4 could find a global optimal in the small-scale, but GGAFAS 1 and GGAFAS 2 sometimes could find only a local optimal. In the middle-scale and large-scale, the GGAFAS 3 could performer best, the second is GGAFAS 4, the third is GGAFAS 1, and the last is GGAFAS 2. From Table 1, we observed the heuristic algorithm 1 could get better solutions, the heuristic 3 is the second, and the heuristic 2 is the poor. We compared GGAFASs with heuristics as Figure 3 shown; most of GGAFASs would get better solutions than heuristics besides GGAFAS 2. The performance of GGAFAS 3 and GGAFAS 4 are excellent in middle-scale and large-scale.. In Table 2, symbol ‘－‘ indicates that the calculation of the final reliability is intractable when n > 20.The data met the exact values shown in Table 1 are underlined, and the most approximate is shown with asterisk (*). The meanings of columns are illustrated as below: 1.. Max_FT : the maximum fitness at the last generation in 30 trials. Min_FT : the minimum fitness at the last generation in 30 trials. FT_Avg : the average fitness at the last generation for 30 trials.. We presented the relative errors of all cases in small-scale in Table 3.. Exact_Max : the maximum of exact value at the last generation in 30 trials. -6-.
(7) Table 2 the experimental data of GGAFASs under rc = 1 and rm = 0.05 GGAFAS 1 Exact_Max. Max_FT. GGAFAS 2. n. m. Min_FT. FT_Avg. Exact_Max. Max_FT. 6. 4. 0.85344. －. －. －. 0.80948. －. －. －. 7. 3. 0.89544. －. －. －. 0.874752. －. －. －. 8. 3. 0.90272. －. －. －. 0.90272. －. －. －. 9. 3. 0.951824. －. －. －. 0.938336. －. －. －. 10. 3. 0.945002. －. －. －. 0.927336. －. －. －. 15. 6. 0.917781. －. －. －. 0.881592. －. －. －. 20. 13. 0.927262. －. －. －. 0.895543. －. －. －. 30. 19. －. 0.919642. 0.862682. 0.8959. －. 0.847595. 0.752405. 0.807822. 50. 22. －. 0.905096. 0.864982. 0.886874. －. 0.847053. 0.74346. 0.796436. 100. 40. －. 0.864804. 0.811001. 0.83788. －. 0.775108. 0.586768. 0.669208. 200. 25. －. 0.949631. 0.909122. 0.937051. －. 0.842965. 0.725463. 0.775536. n. m. Exact_Max. 6. 4. 0.85344. －. －. －. 0.85344. －. －. －. 7. 3. 0.89544. －. －. －. 0.89544. －. －. －. 8. 3. 0.90272. －. －. －. 0.90272. －. －. －. 9. 3. 0.953907. －. －. －. 0.953907. －. －. －. 10. 3. 0.945002. －. －. －. 0.945002. －. －. －. 15. 6. 0.939392*. －. －. －. 0.927822. －. －. －. 20. 13. 0.951664*. －. －. －. 0.933935. －. －. －. 30. 19. －. 0.955008*. 0.936481. 0.945964. －. 0.931501. 0.905442. 0.919903. GGAFAS 3 Max_FT. Min_FT. FT_Avg. GGAFAS 4. Min_FT. FT_Avg. Exact_Max. Max_FT. Min_FT. FT_Avg. 50. 22. －. 0.947798*. 0.929994. 0.940892. －. 0.923432. 0.89685. 0.909386. 100. 40. －. 0.919355*. 0.897167. 0.90743. －. 0.882147. 0.850134. 0.866312. 200. 25. －. 0.976545*. 0.970773. 0.973711. －. 0.966712. 0.932327. 0.952358. HA 1. HA 2. HA 3. GGAFAS 1. GGAFAS 2. GGAFAS 3. Figure 3 the experimental results of each GGAFAS compare with each EA. Table 3 the relative errors of all cases in small-scale.. GGAFAS 4. Fitness 1 0.99 0.98 0.97 0.96 0.95 0.94 0.93 0.92 0.91 0.9 0.89 0.88 0.87 0.86 0.85 0.84 0.83 0.82 0.81 0.8 0.79 0.78 0.77 0.76 0.75 6-4. 7-3. 8-3. 9-3 10-3 15-6. 2013. 3019. 5022. 100- 20040 25. err (%). 6-4. 7-3. 8-3. 9-3. 10-3. HA 1. -3.03. -2.71. -1.39. 0. 0. HA 2. -8.14. -8.69. -5.01. -1.71. -1.23. HA 3. -6.43. -2.61. -1.63. -1.92. -1.97. GGAFAS1. 0. 0. 0. -0.22. 0. GGAFAS 2. -5.15. -2.13. 0. -1.63. -1.87. GGAFAS3. 0. 0. 0. 0. 0. GGAFAS 4. 0. 0. 0. 0. 0. Case. 5. Conclusion -7-.
(8) “Genetic-Algorithm-Based Reliability Optimization for Computer Network Expansion”, IEEE Trans. Reliability, Vol. 44. No1, 1995 Mar, pp.63-72.. In this paper, we used the concept of grouping genetic algorithm to solve the file assignment in a star-topology network problem subject to constraints of nodes’ capacity and files’ copies. We have shown that this problem belongs to NP-hard. In the proposed algorithms, we developed the repair procedure to tune those infeasible chromosomes to feasible ones. We also have compared our proposed algorithms with exact and heuristic algorithms. The exact algorithm usually can find the exact solution in the small-scale; however, it takes much execution time. The heuristic algorithms generally take less execution time than GGAFASs, but it is only able to find out a near-optimal in the middle-scale and large-scale. Experimental results have shown that GGAFASs led to better solutions than heuristics in almost all cases.. [10]. A. E. Smith and D. W. Coit, “Reliability Optimization of Series-Parallel Systems Using a Genetic Algorithm”, IEEE Trans. Reliability, Vol. 45. No. 2, 1996 Jun, pp.254-260.. [11]. C. C. Chiu, R. S. Chen, and Y. S. Yeh, “A Genetic Algorithm for K-node Set Reliability Optimization with Capacity Constraint of a Distributed System”, Proc. Natl. Sci. Counc. ROC(A), Vol. 25. No. 1, 2001, pp.27-34.. [12]. G. Legault and S. Pierre, “A Genetic Algorithm for Designing Distributed Computer Network Topologies”, IEEE Trans. System, Man, and Cybernetics, Vol. 28. No.2, 1998 Apr, pp.249-258.. [13]. A. Kumar, R. M. Pathak, and Y. P. Gupta, “Genetic Algorithm Based Approach for File Allocation on Distributed Systems”, Computer Ops. Res, Vol. 22. No 1, 1995, pp.41-54.. 6. Reference [1]. R. Ramakumar, Engineering Reliability: Fundamentals and Applications, Prentice-Hall, New Jersey, 1993.. [2]. D. J. Chen, K. L. Ku, M. S. Chang, and M. S. Lin, “The Distributed Program Reliability on a Star Topology: Efficient Algorithm and Approximate Solution”, Computers & Operations Research, Vol. 27, 2000,pp. 129-142.. [14]. J. S. Provan and M. O. Ball, “Disjoint products and efficient computation of reliability”, Operations Research, Vol.26, No. 5, 1988, pp.703-715.. D. J. Chen, K. M. Kavi, and P. Y. Chang, “Multimedia File Allocation on VC Networks Using Multipath Routing”, IEEE Trans. Computers, Vol. 49. No. 9, 2000 Sep, pp.971-977.. [15]. D.S. Johnson and M. R. Garey, Computer and Intractability: A Guide to the Theory of NP-completeness, Freeman, San Francisco, 1979.. [16]. M. Gen, and R. Cheng, Genetic Algorithms and Engineering Design, John Wiley & Sons, New York, 1997. [17]. Z. Michalewicz, Genetic Algorithms + Data Structures = Evolution Programs, 3rd eds., Springer-Verlag, New York, 1999.. [18]. D. E. Goldberg, Genetic Algorithms in Search, Optimization, and Machine Learning, Addison-Wesley, 1989.. [19]. C. L. Hwang, F. A. Tillman, V. R. Prasad, and W. Kuo, Optimal Reliability Design Fundamentals and Applications, Cambridge, 2001.. [20]. E. Falkenauer, “A hybrid grouping genetic algorithm for bin packing”, Journal of Heuristics, 2(1), Boston, 1996, pp5-30.. [21]. C. J. Colburn, The Combinatorics of Network Reliability, Oxford, 1987, pp9-11.. [22]. D. R. Shier, Network Reliability and Algebraic Structures, Oxford, 1991, pp8-10.. [3]. [4]. [5]. [6]. [7]. [8]. [9]. G. J. Hwang and S. S. Tseng, “A Heuristic Task Assignment Algorithm to Maximize Reliability of a Distributed System”, IEEE Trans. Reliability, Vol. 42. No. 3, 1993 Sep, pp.408-415. C. S. Raghavendra, S. Hariri and V. K. P. Kumar, “Reliability Analysis in Distributed Systems”, IEEE Trans. Reliability, Vol. 37. No3, 1988 Mar, pp.352-258. J. R. Kim and M. Gen, “GA-based Reliability Design: State-of-the-Art Survey”, Computer and Industrial Engineering, 37, 1999, pp.151-155. J. Campbell and L. Painton, “Genetic Algorithm in Optimization of System Reliability”, IEEE Trans. Reliability, Vol. 44. No. 2, 1995 Jun, pp.172-178. S. T. Cheng, “Topological Optimization of a Reliable Communication Network”, IEEE Trans. Reliability, Vol. 47. No. 3, 1998 Sep, pp.225-233. A. Kumar, R. M. Pathak, and Y. P.Gupta, -8-.
(9) -9-.
(10)
相關文件
Microphone and 600 ohm line conduits shall be mechanically and electrically connected to receptacle boxes and electrically grounded to the audio system ground point.. Lines in
For better efficiency of parallel and distributed computing, Apache Hadoop distributes the imported data randomly on data nodes.. This mechanism provides some advantages
Selcuk Candan, ”GMP: Distributed Geographic Multicast Routing in Wireless Sensor Networks,” IEEE International Conference on Distributed Computing Systems,
Our preliminary analysis and experimental results of the proposed method on mapping data to logical grid nodes show improvement of communication costs and conduce to better
The proposed algorithms use the optimal-searching technique of genetic algorithm (GA) to get an efficient scheduling solution in grid computing environment and adapt to
Hwang, “An O (nk)-time algorithm for computing the reliability of a circular consecutive-k-out-of-n:F system,” IEEE Trans on Reliability, Vol.. Shantikumar, “Recursive algorithm
This study combined the concepts of service science, orienteering problem, geographic information systems and genetic algorithms to design the feeder bus route.. In
After finishing the construct of knowledge management effectiveness model, the empirical study reveal the validity and reliability of the measuring model of present study, and that
