Chapter 1 Introduction
1.3 Thesis organization
The rest of this paper is organized as follows. In Chapter 2, we introduce an existing load management procedure. In Chapter 3, we review existing MMOG architectures. We present the proposed hybrid P2P MMOG architecture and a two-level load management scheme in detail in Chapter 4. In Chapter 5, we discuss simulation results and compare the proposed architecture with the multi-server architecture in terms of response time and deadline miss ratio. In Chapter 6, we give concluding remarks and outline future work.
Chapter 2 Background
2.1 Load distribution techniques
Today’s MMOG’s used three main techniques, zoning, replication, and instancing, to serve hundreds to thousands of player, as shown in Figure 1. Through zoning, the virtual world was divided into smaller blocks which are handled by different servers. Replication takes place when the load of a zone is heavy because a huge number of players center on the zone. The players still can see each other. Finally, instancing is a simplification of replication, which distributes the session load by starting multiple parallel instances of the highly populated zones. The instances are completely independent of each other, meaning that two players from different instances will not see each other, even if they are located at nearby coordinates [15].
Zoning
Replication
Figure 1: Load distribution techniques [27].
2.2 Load management in MMOGs
In [29], the authors proposed a new predictive resource provisioning method based on a stack of five services depicted in Figure 2. Firstly, a monitoring service collects online metrics related to the performance of an MMOG session which could be of two kinds: (1) game session-related, such as the number of entities and their positions in the game world, and (2) resource-related, such as the CPU, memory, storage, and network load. Secondly, a load prediction service is used to anticipating the future game world entity distribution from historical traces. A capacity planning service includes generic analytical models to map an entity distribution into a resource’s load, such as processor, memory, storage or network load.
In this service, it focuses on foreseeing potential hotspots in the game servers which make the game environment fragmented and unplayable. A resource allocation service is provisioning of a right amount of resources required for a proper execution that guarantees a good
Figure 2: Load management cycle [29].
Chapter 3
Related Work
3.1 Client-server MMOG architecture
Client-server MMOG architecture has players (clients) that send requests to a single server. It is simple and easy to implement, but it is less scalable. As shown in Figure 3, commands from players must go through a single server, and the single server handles commands and returns state updates to players. The single server must be in the presence of insufficient bandwidth and copious waiting time for players. To resolve the problems, there are several approaches [28] to distribute traffic among multiple servers, such as a mirrored server [29], generic proxy [30], or booster box [31].
Server
Player
Request State Update
Figure 3: Client-server MMOG architecture [34].
3.2 Multi-server MMOG architecture
Multi-server MMOG architecture [19] has various kinds of servers which provide different functionalities as shown in Figure 4. The various servers are described as follows [19]:
Master Server - This server handles access of players and communicates with operated servers. It assists players logging in a zone server and transfers data of players to the corresponding zone server from the character server.
Character Server - This is mainly stored the data of players. It assists players in using the same characters on any zone servers.
World Daemon - This server transfers players from a zone server to another zone server that players expected. It also monitors each load of zone servers.
Zone Cluster Server – It is composed of zone servers and mainly serves players. A zone server is assigned to serve a specific zone.
Client - A client moves between zone servers. A client sends a request to an interested zone server.
In this MMOG multi-server architecture, the players in the same zone connect to the specific zone server. Some zone servers will suffer heavy load if many players move into the same zone. Hence, load balancing is needed to prevent the downgrade of service. In [29], it creates a management server that monitors zone servers and decides a load balancing strategy.
In this case, the management server would become the bottleneck of the system [20].
3.3 P2P MMOG architecture
In P2P MMOGs, they advance to multi-server architecture by lowering the cost of centralized infrastructures and by distributing the processing load [20]. But the security is their drawback. They have to request clients to share loading and clients could get some data of other clients. Because of this reason, the P2P MMOG architecture usually runs on lower security games.
As shown in Figure 5 [20], the server serves players and assigns some normal nodes to become service nodes. A normal node is a player. When the server loading is more than a threshold, the server will transform some normal nodes to service other players. This process could cut down server loading but the security emerges more problems.
Master Server
World Daemon World Daemon
Character Server
Zone Cluster Server Zone Cluster Server Zone Cluster Server Zone Cluster Server
Figure 4: Torque MMOG multi-server architecture [19].
3.4 P2P cloud computing architecture
As shown in Figure 6 [26], the architecture includes three basic roles: User, Central Peer and Side Peer. The Central Peer and Side Peer form two P2P networks, called central P2P network and side P2P network, respectively. The central P2P network mainly maintains metadata of dynamic mapreduce and backups DHT P2P storage cloud. The side P2P network is mainly used to provide storage and computing resources.
C3 C2
Upload/Download Data
Side
Figure 5: P2P MMOG architecture [20].
Chapter 4
Proposed Hybrid P2P MMOG Cloud Architecture
In this chapter, we propose a hybrid P2P cloud architecture for MMOGs which includes two-level load management: (1) multi-threshold load management for each game server and (2) load management among game servers. The components of the proposed architecture will be described in section 4.1. Game servers in the game server cloud share information by P2P manner. Section 4.2 introduces a game server cloud. In order to support this architecture, we propose a multi-threshold load management procedure in section 4.3.
4.1 Proposed cloud architecture
MMOG environments require a high degree of flexibility to respond to environmental changes, including load change. Thus, we combine cloud computing with MMOGs to increase flexibility of resource allocation. Using virtualization technical on cloud could help us get more security and scalability. The proposed cloud architecture includes four basic components: game server cloud, players, character database and game regions data storage, as shown in Figure 7. We introduce each component’s characteristics as follows:
Game servers cloud – It consists a number of game servers and a game server has a number of virtual machines (VMs) to serve players. A game server receives requests from players, calculates new game states in regions, and responses to players. A game server is also in charge of creating accounts for players when they login for the first time.
Players – They move between game regions of a game world. The load of a game region increases when players move into the game region.
Character database – It stores the data of players, such as equipment, rank, site, etc. It also backups players’ data. The backup method is based on backup method of hadoop.
Game regions data storage – It maintains game states in each region. It also backups game states and object states in each game region.
4.2 Game servers cloud
In MMOG environments, there are huge messages, especially communication messages between players. Game servers use a P2P protocol to exchange load information and game region data. Game servers also use the P2P protocol to exchange the information of players.
By P2P flooding, each game server can know the other game servers’ load. When a game server is overloaded, it will migrate some players to other game servers by the proposed game server load management procedure.
Figure 7: Hybrid P2P MMOG cloud architecture.
4.3 Multi-threshold load management
In the game servers cloud, we allocate VMs from game servers to service game regions.
Game regions do not setup on specific virtual machines. In this way, it could help the system to get high scalability. We do not use a single master server in the game servers cloud, Game servers communication is based on a P2P protocol. Game servers can exchange load information by flooding in the P2P protocol. Game servers can perform load management by themselves, and a game server is also in charge of creating accounts for players when they login for the first time. This way can help resource allocation more efficient.
The multi-threshold load management focuses on VMs management. We use four thresholds (T1, T2, T3, and T4) to define five loading layers (L1, L2, L3, L4, and L5) for each VM.
T1 is the lower bound of VM load. T2 and T3 define two preferred loading layers to service players. T4 is the upper bound of VM load. As shown in Figure 8, each loading layer represents a different load and status.
– L1 (Light): For a VMi operating in this layer, its players will be migrated to another available VMs in these layers; If there are no VMs in these layers, the game server load management will be executed.
Figure 9 and Figure 10 show the flow chart of the multi-threshold load management algorithm. The details of the multi-threshold load management algorithm are described in the following.
1) We monitor the VMs in each game region and sort the VMs in each game region according to their loadings in an ascending order.
2) If the VM in L5 is empty, go to step 7. Otherwise, go to step 3.
3) In this step, we know some VMs exist in L5. We check if there are VMs in L2. If there are VMs in L2, we migrate VMi with maximal load in L5 to the VMj with minimal load in L2, and then go to step 2. Otherwise, go to step 4.
4) We check if there are VMs in L3. If there are VMs in L3, we migrate VMi with minimal load in L5 to the VMj with minimal load in L3, and then go to step 2.
Otherwise, go to step 5.
5) We check if there are VMs in L1. If there are VMs in L1, we migrate VMi with minimal load in L5 to the VMj with minimal load in L1, and then go to step 2.
L3 (Medium) L5 (Heavy)
L1 (Light) L2 (Low) L4 (High) T4
T1
T2
T3
Figure 8: Multi-threshold load management.
6) Executing the VM creating procedure to create a VM to share the load in this game
Figure 9: Multi-threshold load management algorithm (1/2).
9) We check if there are VMs in L2. If there are VMs in L2, we migrate VMi with minimal load in L1 to the VMj with maximal load in L2, release VMi, and then go to step 7. Otherwise, go to step 10.
10) We check if there are VMs in L4. If there are VMs in L4, we migrate VMi with minimal load in L1 to the VMj with maximal load in L4, release VMi, and then go to step 7. Otherwise, exit the procedure.
When a game region needs a new VM, the VM creating procedure is executed, as shown in Figure 11. Firstly, the game server checks its resource. If it has enough resources, the game server will create a new VM to service the game region. If not, the game server will turn down the VM creating request. The game server executes the game server load management procedure to transfer players to another game server.
A
Figure 10: Multi-threshold local load management algorithm (2/2).
In the game server load management procedure, we mainly consider the physical distance between players and game servers, because players need to be served with low response time. As shown in Figure 12, firstly we find a neighboring game server who is serving the same game region and has the lowest load. If not available, we select a neighboring game server that is not serving the same game region and has the lowest load. If not available, we select a distant game server that has capacity to service more players. The selected game server will service the migrated players. Finally, if we could not find a game server who has capacity to service more players, then it means all game servers are overloading.
Start
Activate the VM to service the
region
Does the game server have enough resource to create a VM
Create a new VM
End
Turn down the VM creating request Yes No
Execute the game server load management
procedure
Figure 11: VM creating procedure.
Transfer some players from calling
game server to this game server
End
Is there a game server who has capacity to game server who is serving the
same game region and has he capacity to service more
players
No
Yes
Is there a neighboring game server who is not serving the same game region and has the capacity to service
more players No
All game servers are overloading
Figure 12: Game server load management procedure.
Chapter 5
Performance Evaluation
In this chapter, we first describe simulation setup and evaluation metrics. Then, we evaluate the proposed hybrid P2P MMOG in terms of average response time, and average deadline miss ratio.
5.1 Simulation setup and evaluation metrics
We use CloudSim [27] to simulate players’ behaviors by sending service requests to game servers. Players’ requests were collected from Stendhal MMORPG using Wireshark.
The Stendhal MMORPG is running on Intel i7 2.93 GHz CPU with 512 MB RAM and 100 Mbps Ethernet. The average service time to serve a request is 0.2 ms. There are three game servers and the upper bound of the response time is set to 300 ms according to [30]. Each game server can create at most 40 VMs. The capability of a VM is 2000 MIPS with 1024 MB RAM and 100 Mbps Ethernet (P3 capability). Each VM can support at most 59 players (obtained from CloudSim). The total number of game regions is 30. In addition, a game region is served by at least one VM, if there are players.
5.1.1 Average response time
The average response time is defined as the elapsed time between the time a player sends a request to the time that a player actually receives the response, which is formulized as follows:
5.1.2 Average deadline miss ratio
The average deadline miss percentage ratio is defined as the response time being more than a real time bound, and we set the real time bound to 300 ms according to [32], which is formulized as follows:
where Davg= average deadline miss ratio
RD = the total number of responses that miss the deadline in a game j
server j where Tavg= average response time
TRi= the time interval for player to receive a response, and i is the ith request.
5.2 Comparison between multi-server architecture and proposed hybrid P2P cloud architecture
Figure 13 shows the average response time of the multi-server architecture and proposed hybrid P2P cloud architecture. The multi-server architecture has the lower response time initially because a regional server’s capacity is higher than a VM’s. When the number of players increases to 3490, the multi-server architecture has higher response time. This is because its regional servers serve specific game regions. As a result, players are queued up for a specific regional server. In our architecture, each VM has lower capacity, but our architecture has higher scalability. VMs can be migrated to a busy game region and the response time of players can be reduced. Because of this, our architecture performs better after the number of players exceeding 3490.
Figure 14 shows the deadline miss ratio between the multi-server architecture and proposed hybrid P2P cloud architecture. In the multi-server architecture, it has the lower deadline miss ratio under lower number of players. It shows that with enough resources in the multi-server architecture, the multi-server architecture can have good performance. The proposed architecture has a smaller deadline miss ratio under medium to high load than the multi-server architecture.
Figure 15 shows the average response time of the proposed hybrid P2P cloud architecture under different thresholds settings. We set different thresholds to T1, T2, T3 and T4. Before the number of players reaches 6400, the 0.2-0.3-0.7-0.8 setting has lower average response time. This is become that new VMs are created to service players under lower load (0.8). After the number of players is over 6400, the average response time for this setting becomes higher than that of the other setting. Because VMs are activated early to service game regions, there may be no VM to service really hotspot game regions later.
Figure 13: Average response time between multi-server and proposed hybrid P2P cloud
800 1600 2400 3200 4000 4800 5600 6400 7200 8000
A v era g e r es po ns e ti m e (m s)
800 1600 2400 3200 4000 4800 5600 6400 7200 8000
D ea dl ine m is s ra ti o (% )
800 1600 2400 3200 4000 4800 5600 6400 7200 8000
D ea dl ine m is s ra ti o (% )
Number of players
Multi-server Hybrid P2P cloud (proposed)
Figure 15: Average response time under different thresholds of load for the proposed architecture.
100 130 160 190 220 250 280 310 340 370 400 430 460
800 1600 2400 3200 4000 4800 5600 6400 7200 8000
A v era g e r es po ns e ti m e (m s)
Number of players
0.1-0.2-0.8-0.9 0.2-0.3-0.7-0.8
Chapter 6 Conclusion
6.1 Concluding remarks
In this paper, we propose a hybrid P2P cloud architecture which includes multi-threshold load management and game server load management for MMOG cloud environments. The multi-threshold load management can make the utilization of virtual machines more efficient.
The game server load management transfers some players from an overloaded game server to a neighbor game server with available capacity. The proposed hybrid P2P cloud architecture can support 10.31% more players under no deadline (300 ms) miss compared to the multi-server architecture. The proposed hybrid P2P cloud architecture can reduce the average response time by 20.6% compared to the multi-server architecture under medium to high load through flexible allocation of resources (virtual machines). The proposed architecture also has a 27.9% smaller deadline miss ratio than the multi-server architecture under medium to high load.
6.2 Future work
We may apply the proposed hybrid P2P cloud architecture to multi-game environments for flexible allocation of resources. We will integrate an efficient load prediction scheme with the proposed hybrid P2P cloud architecture to achieve more efficient allocation of resources so as to further reduce the deadline miss ratio and average response time.
Bibliography
[1] G. Dolbier and A. Goldschmidt, “The business of interactive entertainment,” in Proceedings of the IBM Digital Media Solutions, May 2006.
[2] E. Cronin, B. Filstrup, and A. Kurc, “A distributed multiplayer game server system,”
Technical Report the University of Michigan, May 2001.
[3] A. Shaikh, S. Sahu, M.-C. Rosu, M. Shea, and D. Saha, “On demand platform for online games,” IBM Systems Journal, vol. 45, no. 1, pp. 7–20, 2006.
[4] B. Hack, M. Morhaime, J.-F. Grollemund, and N. Bradford, “Introduction to vivendi games,” [Online] Available: http://www.vivendi.com/, Jun 2006.
[5] S. Sukhyun, et al., "Blue Eyes: Scalable and reliable system management for cloud computing," in Proceedings of the IEEE International Symposium on Parallel &
Distributed Processing, pp. 1-8, 2009.
[6] C.Majewski, C. Griwodz, and P. Halvorsen, “Translating latency requirements into resource requirements for game traffic,” in Proceedings of the International Network, pp.
113-120, 2006.
[7] J. Slegers, I. Mitrani, and N. Thomas, "Evaluating the optimal server allocation policy for clusters with on/off sources," the Performance Evaluation, vol. 66, pp. 453-467, 2009.
[8] R. Stanojevic and R. Shorten, "Load balancing vs. distributed rate limiting: an unifying framework for cloud control," in Proceedings of the IEEE International Conference on Communications, pp. 1-6, 2009.
[9] W. Streitberger and T. Eymann, "A simulation of an economic, self-organising resource allocation approach for application layer networks," the Computer Networks, vol. 53, pp.
1760-1770, 2009.
[10] Y. Lai and S. ZhongZhi, "An efficient data mining framework on Hadoop using Java persistence API," in Proceedings of the IEEE 10th International Conference on Computer and Information Technology, pp. 203-209, 2010.
[11] Z. Liang-Jie and Z. Qun, "CCOA: Cloud Computing Open Architecture," in Proceedings of the IEEE International Conference on Web Services, pp. 607-616, 2009.
[12] V. Nae, A. Iosup, and R. Prodan, "Dynamic resource provisioning in massively multiplayer online games," IEEE Transactions on Parallel and Distributed Systems, vol.
PP, pp. 1-1, 2010.
[13] Torque MMO Kit - Server Architecture
http://www.mmoworkshop.com/trac/mom/wiki/ServerArchitecture
[14] A. E. Rhalibi and M. Merabti, "Interest management and scalability issues in P2P MMOG," in Proceedings of the Consumer Communications and Networking Conference, pp. 1188-1192, 2006.
[15] X.-B. Shi, D. Yang, L. Du, and Z.-W. Wang, "Research on service management techniques for P2P MMOG," in Proceedings of the International Conference on Internet Technology and Applications, pp. 1-4, 2010.
[16] C. Yang, W. Tianyu, and L. Jianxin, "An efficient resource management system for
[16] C. Yang, W. Tianyu, and L. Jianxin, "An efficient resource management system for