Case Study: MagePowerCraft
5.3 Validations, Experiments and Results
5.3.1 TABS Validations
We validated TABS in three cases with different settings. For each case, We assigned the same character set to the two teams. Obviously, the ability power of the both teams are balanced. Thus, the balanced conclusion should be drawn by TABS, i.e., the ϵavgvalue of each case should all be small. The two teams in the three cases are:
1. (Team 1: Fire Wizard) verses (Team 2: Fire Wizard)
2. (Team 1: Fire Wizard + Ice Wizard) verses (Team 2: Fire Wizard + Ice Wizard) 3. (Team 1: Fire Wizard + Ice Wizard + Priest) verses (Team 2: Fire Wizard + Ice
Wizard + Priest)
Training Model: GA
The following tables show the validation results of the three cases by applying GA:
Validation: Case 1
Game Set 1 2 3 4 5
Team1:Team2 491:509 490:510 483:517 474:526 527:473
Game Set 6 7 8 9 10
Team1:Team2 510:490 497:503 499:501 492:508 491:509
Average 495.4:504.6 Error ϵavg 0.92%
Table 5.3: The results of Validation by GA, Case 1.
Validation: Case 2
Game Set 1 2 3 4 5 Team1:Team2 518:482 488:512 497:503 523:477 507:493
Game Set 6 7 8 9 10
Team1:Team2 482:518 484:516 490:510 527:473 477:523
Average 499.3:500.7 Error ϵavg 0.14%
Table 5.4: The results of Validation by GA, Case 2.
Validation: Case 3
Game Set 1 2 3 4 5
Team1:Team2 515:485 520:480 521:479 487:513 487:513
Game Set 6 7 8 9 10
Team1:Team2 490:510 496:504 459:541 480:520 488:512
Average 494.3:505.7 Error ϵavg 1.14%
Table 5.5: The results of Validation by GA, Case 3.
Training Model: PSO
The tables below give the validation results of the three cases by applying PSO:
Validation: Case 1
Game Set 1 2 3 4 5
Team1:Team2 495:505 507:493 499:501 498:502 498:502
Game Set 6 7 8 9 10
Team1:Team2 490:510 515:485 521:479 480:520 516:484
Average 501.9:498.1 Error ϵavg 0.38%
Table 5.6: The results of Validation by PSO, Case 1.
Validation: Case 2
Game Set 1 2 3 4 5 Team1:Team2 508:492 508:492 498:502 518:482 504:496
Game Set 6 7 8 9 10
Team1:Team2 508:492 484:516 492:508 468:532 507:493
Average 499.5:500.7 Error ϵavg 0.1%
Table 5.7: The results of Validation by PSO, Case 2.
Validation: Case 3
Game Set 1 2 3 4 5
Team1:Team2 507:493 515:485 510:490 489:511 502:498
Game Set 6 7 8 9 10
Team1:Team2 516:484 496:504 510:490 508:492 489:511
Average 504.2:495.8 Error ϵavg 0.84%
Table 5.8: The results of Validation by PSO, Case 3.
Consider that GA was used, the average balance error values, ϵavg, of the three cases are 0.92%, 0.14% and 1.14%, respectively. Thus, the system error value by using GA is the largest value among them, i.e. ϵGAT ABS = 1.14%. And for PSO, the average balance error values of the three cases are 0.38%, 0.1% and 0.84%, respectively. Hence, the system error value by using PSO ϵP SOT ABS = 0.84%.
The results show that the validity of TABS to MagePowerCraft by both training mod-els is satisfactory. The two system errors, ϵGAT ABS and ϵP SOT ABS, are for determining whether or not the ability power of two teams are similar in MagePowerCraft by using the corre-sponding training model. That is to say, if the error of a testing result is lower than the corresponding system error of the training model, we conclude that the ability power of the two teams is balanced. Otherwise, the two teams are judged as unbalanced. Notice that the system error values are subjected to change for different games with different features.
5.3.2 Experiments
We take the role as game designers and demonstrate three independent experiments of using TABS to tune the skill settings of the character classes in MagePowerCraft (See Appendix A). The goal of our experiments are to adjust the skill parameters and make the skill ability power of one team matching with the other in each case, i.e. the final ϵavg of each case should be smaller than the corresponding system error, ϵGAT ABS and ϵP SOT ABS, which we had obtained earlier from the validations in Section 5.3.1. Each experiment is done by GA first and then repeated again by using PSO The team combanition of the three experiments are:
1. (Team 1: Fire Wizard) verses (Team 2: Ice Wizard)
2. (Team 1: Fire Wizard + Shaman) verses (Team 2: Ice Wizard + Priest) 3. (Team 1: Fire Wizard) verses (Team 2: Ice Wizard + Priest + Shaman)
Experiment 1:
In this experiment, we would like to show the character-to-character level balancing.
Before adjusting the skill parameters, we need to evaluate the balance situation of the two characters, and below are the results with the original skill settings by using GA and PSO
Game Set 1 2 3 4 5
Team1:Team2 615:385 620:380 648:352 586:414 597:403
Game Set 6 7 8 9 10
Team1:Team2 627:373 632:368 623:377 583:417 633:367
Average 616.4:383.6 Error ϵavg 23.28%
Table 5.9: The results of Experiment 1, with original skill settings by applying GA
Game Set 1 2 3 4 5 Team1:Team2 594:406 613:387 627:373 635:365 656:344
Game Set 6 7 8 9 10
Team1:Team2 673:327 657:343 602:398 606:394 648:352
Average 631.1:368.9 Error ϵavg 26.22%
Table 5.10: The results of Experiment 1, with original skill settings by applying PSO
Apparently, the two teams were not balanced (23.28% > ϵGAT ABS= 1.14% and 26.22%
> ϵP SOT ABS= 0.84%). Based on the scores, we find out that Team 1 was more powerful than Team 2. To balance the ability power of two teams, as an example, we choose to strengthen the power of Freezing Sword and Blizzard of the Ice Wizard in Team 2, while we weaken the Fire Ball skill of Fire Wizard in Team 1 at the same time. It took several steps for us to fine-tune the skill parameters. Once a new setting is made, we run it with TABS and see if the new setting is balanced. And after a few tries, we eventually came up with the setting the (Skill Power,Extra Skill Point) of the skill Freezing Sword, Blizzard and Fire Ball as (244%, 556), (540%, 574) and (322%, 360), respectively. This new setting gives the balanced results as listed in Table 5.11 and 5.12 for G.A and PSO
Game Set 1 2 3 4 5
Team1:Team2 500:500 512:488 511:489 534:466 480:520
Game Set 6 7 8 9 10
Team1:Team2 505:495 501:499 483:517 516:484 487:513
Average 502.9:497.1 Error ϵavg 0.58%
Table 5.11: The results of Experiment 1, with new skill settings by applying GA
Game Set 1 2 3 4 5 Team1:Team2 501:499 493:507 514:486 518:482 491:509
Game Set 6 7 8 9 10
Team1:Team2 500:500 500:500 508:492 512:488 501:499
Average 503.8:496.2 Error ϵavg 0.76%
Table 5.12: The results of Experiment 1, with new skill settings by applying PSO
Experiment 2:
In this experiment, we would like to show the team-to-team level balancing. We evaluated the balance situation of the two teams by TABS with the original settings, and down below are the results.
Game Set 1 2 3 4 5
Team1:Team2 515:485 491:509 429:571 497:503 491:509
Game Set 6 7 8 9 10
Team1:Team2 471:529 475:525 499:501 490:510 474:526
Average 483.2:516.8 Error ϵavg 3.36%
Table 5.13: The results of Experiment 2, with original skill settings by applying GA
Game Set 1 2 3 4 5
Team1:Team2 461:539 452:548 516:484 455:545 458:542
Game Set 6 7 8 9 10
Team1:Team2 458:542 459:541 462:538 497:503 467:533
Average 469.4:530.6 Error ϵavg 6.12%
Table 5.14: The results of Experiment 2, with original skill settings by applying PSO
The error shows that the two teams were not balanced (3.36% > ϵGAT ABS = 1.14% and 6.12% > ϵP SOT ABS = 0.84%). After checking the gaming statistics of Shaman recorded by TABS as shown in Figure 5.5, we noticed that the average score per cast of skill Soul Blast
Figure 5.5: The gaming statistics of Shaman with the origin skill setting.
by Shaman in Team 1 is relatively low. Since a low average score per cast could indicate that the power of a skill may be too low. Therefore, we decided to enhance the attack power of Soul Blast in this experiment. This time we quickly found the balanced setting for only one attempt of trying, which is to set the (Skill Power,Extra Skill Point, Max Hit) of (Soul Blast) from (171%, 274, 5) to (385%, 314, 6). The results with the new settings are listed below. Figure 5.6 shows the gaming statistics of Shaman after the new settings.
Game Set 1 2 3 4 5
Team1:Team2 477:523 472:528 494:506 499:501 507:493
Game Set 6 7 8 9 10
Team1:Team2 487:513 526:474 516:484 494:506 492:508
Average 496.4:503.6 Error ϵavg 0.72%
Table 5.15: The results of Experiment 2, with new skill settings by applying GA
Game Set 1 2 3 4 5
Team1:Team2 469:531 492:508 526:474 488:512 508:492
Game Set 6 7 8 9 10
Team1:Team2 519:481 503:497 524:476 467:533 498:502
Average 499.4:500.6 Error ϵavg 0.12%
Table 5.16: The results of Experiment 2, with new skill settings by applying PSO
Figure 5.6: The gaming statistics of Shaman with the new skill setting.
Experiment 3:
In the last experiment, we would like to show the character-team balancing. We may assume that this experiment is the situation of trying to balance between a single non-player-controlled character (NPC) boss, the Fire Wizard, and a team of player characters.
As usual, we evaluate the balance situation of the two sides first. The results of the eval-uations are as follows.
Game Set 1 2 3 4 5
Team1:Team2 515:485 491:509 429:571 497:503 491:509
Game Set 6 7 8 9 10
Team1:Team2 471:529 475:525 499:501 490:510 474:526
Average 483.2:516.8 Error ϵavg 3.36%
Table 5.17: The results of Experiment 3, with original skill settings by applying GA
Game Set 1 2 3 4 5
Team1:Team2 461:539 452:548 516:484 455:545 458:542
Game Set 6 7 8 9 10
Team1:Team2 458:542 459:541 462:538 497:503 467:533
Average 469.4:530.6 Error ϵavg 6.12%
Table 5.18: The results of Experiment 3, with original skill settings by applying PSO
Team 1 with only one character is obviously weaker than Team 2 which has three characters. Because that most NPC boss has high HP value than the player characters, thus, this time we will adjust the maximum HP of the Fire Wizard and letting it to match the power ability of the team of three players. After several adjustments, the two teams are finally balanced as the result listed in Table 5.19 and 5.20 with setting the maximum HP of the Fire Wizard as 55393.
Game Set 1 2 3 4 5
Team1:Team2 518:482 490:510 505:495 525:475 485:515
Game Set 6 7 8 9 10
Team1:Team2 503:497 487:513 513:487 511:489 492:508
Average 502.9:497.1 Error ϵavg 0.58%
Table 5.19: The results of Experiment 3, with new skill settings by applying GA
Game Set 1 2 3 4 5
Team1:Team2 513:487 506:494 491:509 504:496 492:508
Game Set 6 7 8 9 10
Team1:Team2 502:498 518:482 511:489 506:494 490:510
Average 503.3:496.7 Error ϵavg 0.66%
Table 5.20: The results of Experiment 3, with new skill settings by applying PSO