Chapter 6 Experiment and Evaluation
6.2. Round 2: Perfomance baseline of DOIT Platforrm
In our platform, we adopt the client-gateway-server architecture. It is desirable to realize the performance baseline under this architecture. In this round, we also use the simple echo program to experiment with the performance baseline of client-gateway-server architecture.
6.2.1. Hardware Configuration
In this round, we prepared one more machine to run as a gateway. This machine has the same configuration as the server.
Usage Number Configuration
Server 1 P4 2.4GHz CPU with 1MB ram
Gateway 1 P4 2.4GHz CPU with 1MB ram
Table 6-3: Hardware configuration. of round 2
6.2.2. Software Configuration
Server Client
Client
Client Gateway
140.113.88.* 192.168.1.*
Figure 6-3: Communication architecture of round 2
Similarly, we designed a client generator program to perform stress test. The client number also scaled from 500 to 4000 between 8 runs. The most different point is that the test environment was separated into 2 LAN. Clients connected to the gateway in a subnet and the gateway connected to server in another one (see Figure 6-3). Therefore, the traffic from client to gateways didn’t inference the one between the gateway and the server.
Concerning the test program, we designed an echo server as well. The difference is that the echo semantic was implemented as a game logic and we deployed it to the game platform. So this program can be considered as the simplest game and the performance can be considered as the performance baseline. Similarly, there were 8 runs and each run lasted for ten minutes.
6.2.3. Experiment Result
Client number Average Response
Time Standard Deviation
500 17.87 43.13
1000 13.09 27.20
1500 12.18 30.28
2000 18.27 34.21
2500 17.83 37.63
3000 20.17 45.72
3500 27.66 121.29
4000 33.22 135.09
Table 6-4: Experiment result of round 2
Round2
0 100 200 300 400 500
0 1000 2000 3000 4000
Client Number
Respoi nse t ime (ms)
Avg.
SD
Figure 6-4: Experiment result of round 2
6.2.4. Evaluation
We observe that the result was very similar to the result of round 1. Although the average response was slightly larger than that of round 1, it was also less than 100 even if the client number reached to 4000. The standard deviation also had the level near 100 ms. It is interesting to note that the average response time in 500 clients was higher than that in 1000 clients. The reason could be caused that it fully exploited the benefit of message aggregation in test of 1000 clients such that it got the better performance than that in 500 clients. The similar result also appeared in round 3.
6.3. Round 3: Simulate a Real Game
In the last round, we tried to simulate the real game environment and observed the performance.
6.3.1. Hardware Configuration
We used two computers for servers, two for gateways, ten for clients. The detail configuration is list as follow.
Usage Number Configuration
Server 2 P4 2.4GHz CPU with 1MB ram
Gateway 2 P4 2.4GHz CPU with 1MB ram
Client 10 P4 1.6~2.4GHz CPU with 512MB ram
Table 6-5: Hardware Configuration of round 3
6.3.2. Software Configuration
AOI AOI
Figure 6-5: The move logic
To simulate the real game world, we implemented the most significant game logic, avatar movement, in the test program. However, the different implementation of the move logic will affect the result obviously. So we should make more effort to describe the implementation detail of move logic. We should first define the term AOI (Area of Interest) as the area in which all events are interesting to a given game object.
This game object is interested in all the game objects in AOI. Then, we explain the move logic by mean of the Figure 6-5. Assume that one avatar moves right, as the figure indicates, we should send the player update message to all the game objects inside the AOI and the region of the oblique line. This is because the game objects in the AOI of new location should be interested in the new state of this avatar. The game objects in the leftmost of the AOI of the old location should receive the player update to indicate that this avatar was move out of their AOI. In addition to send updates to the surrounding objects, the avatar also needs to extend his eye sight. So the states of the game objects in the oblique line area should also be sent back to avatar.
Server
Client Gateway
Gateway Server
Client Client Client Client
Client Client Client Client Client
140.113.88.* 192.168.1.*
Figure 6-6: Communication architecture of round 3
The Figure 6-6 depicts the communication architecture of this round. The virtual world was divided into 4 equal size regions plus a login region. They were deployed to 2 game servers. In addition, we provided two gateways for clients to connect in.
Similarly, we put the two servers in the private network. The gateway was responsible for routing messages between internal and external network. Due to multiple regions, it is possible that one avatar will migrate from one region to another, the avatar migration also be implemented in this round.
Concerning the test program, there were also 10 machines running the client generator program. We separated them into two groups. Machines in the same group connected to the same gateway. There were 8 runs in this round. The number of clients is increase with the number of 50 in each client generator program. Therefore, the gateway had 2000 clients connected concurrent at most, and the game platform had 4000 clients in the virtual world concurrently. Furthermore, we performed different tests for different map size and AOI size. The map had 500 x 500 and 1000 x 1000 two different sizes, and AOI had 9 x 9 and 16 x 16 two different sizes. We hope to realize the performance in different environment. Besides, in each test, we pick the
updates with avatar migration to do further analysis.
6.3.3. Experiment Result
Average Response Time - Total 500x500
Table 6-6: Average Response Time – Total
Average Response Time - Total
0.00
0 1000 2000 3000 4000
Client Number
Figure 6-7: Average response time – total
Standard Deviation - Total
Table 6-7: standard deviation – total
Standard Deviation - Total
0 1000 2000 3000 4000
Client Number
Figure 6-8: Standard deviation total
Average Response Time - Migration
Table 6-8: Average response time – migration
Average Response Time - Avatar Migration
0.00
0 1000 2000 3000 4000
Client Number
Figure 6-9: Average response time – migration
Standard Deviation - Migration
Table 6-9: Standard Deviation – Migration
Standard Deviation - Avatar Migration
0.00
0 1000 2000 3000 4000
Client Number
Figure 6-10: Standard Deviation – Migration
500x500 9x9 1000x1000 9x9 500x500 16x16 1000x1000 16x16 Gateway Server Gateway Server Gateway Server Gateway Server
500 6 2 2 2 2 2 5 2
Table 6-10: CPU load
500x500 9x9 1000x1000 9x9 500x500 16x16 1000x1000 16x16 T o
Table 6-11: The amount of commands forwarded per second in a gateway
500x500 9x9 1000x1000 9x9 500x500 16x16 1000x1000 16x16
500 580 525 647 548
Table 6-12: The total amount of messages forwarded per second in a gateway
0
0 1000 2000 3000 4000 5000
500x500 9x9 1000x1000 9x9 500x500 16x16 1000x1000 16x16
Figure 6-11: The total amount of messages forwarded per second in a gateway
6.3.4. Evaluation
In Figure 6-7 and Figure 6-8 we can observe that different map size and different AOI size will lead to obviously different result. In all the tests, we find that the data of map size 500 x 500 and AOI 16 x 16 got the worst result. We think that the result can be expected because it have the largest density in this configuration. In this configuration, the average response time grows steadily under 2500 clients concurrently. But it starts to have dramatic growth after 2500 clients. Concerning the two configuration of 1000 x 1000, 16 x 16 and 500 x 500, 9 x 9, there is no obvious growth until the client number reaches to 3500 clients. But the amount of growth is steady and slow. Concerning the configuration of 1000 x 1000, 9 x 9, the average responses time is almost under good result throughout the test.
If we observe the percentage of CPU usage in Table 6-10, we can find that the performance bottleneck is on the gateway. The CPU usage is always under 50% in the servers. We take a future look. We recorded the update sent per second in the gateway
at the Table 6-11. We consider it have tight relation to the performance. The main game logic is move message. According to the move game logic described above, the number of update messages should be a function of total clients, map size, aoi size, and clients in the gateway. We first consider how many update messages are sent when one move message is process. First of all, the update message should be sent back to the avatar himself. Furthermore, the updates from surroundings and updates to surroundings can totally seem as function of the avatar in his AOI. It can calculate by
map aoi ntotal
×
, where ntotal
is the total client number and map and aoi are map size and
AOI size respectively. So a move message can cause
) messages. Finally, the client sent one move message per second. So the updates per second can be described as the following formula
gat total
n map aoi
update=(n × +1)× , where ngat is the number of clients on a gateway
More concisely, we use d as the number of game objects in an aoi, then the updates per second is described as following
ngat
d
update=( +1)× , where aoi map d = ntotal ×
and the total messages the gateways processed is
gat
Therefore, we calculate the prediction number of update messages processed by gateway according the last formula and list it in the “Prediction” column for each configuration in Table 6-11. The prediction data is very close to the actual data. The actual data is slightly more than prediction data. The reason could be that there were some avatar migrations occurred and the avatar migration may cause much more updates. Finally, we calculate the total messages per second processed by a gateway
by the given update messages per second plus the command message sent by clients.
The result is listed in Table 6-12 and depicted in Figure 6-11. According the result we get above and map it to this diagram, we can find that a gateway can bear 5000~6000 messages per second.
Finally, we observe the impact of avatar migration in Figure 6-9 and Figure 6-10.
The trend of updates of avatar migration is very similar to the general message updates. The average response time is 1.5 times larger than general message. But it is acceptable because the avatar migration does not happens often.
6.4. Summary
To evaluate our platform, we use three rounds to experiment our platform. In the first round, we hope to realize the performance of underlying NetEngine. Until the client numbers reach to 4000, the average response time also have the level of less than 100 ms. But there is steep rise of standard deviation when the client number reach to 4000. In the second round, we try to find the performance baseline of client-gateway-architecture. The result is very similar to round 1 expect that the average response time is slightly larger than that of round1. So the client-gateway-server architecture doesn’t downgrade the performance very much. In round 3, we simulate the real game world. We implement the most significant game logic, avatar movement, in our experiment platform. We found that the performance bottleneck lay on the gateways. Furthermore, we observed the amount of update messages forwarded by gateway. We construct a formula to calculate the number of updates for one command. The result is that the gateway can bear about 5000~6000 messages per second.
Chapter 7 Future Works and Conclusions
7.1. Conclusions
To craft a MMOG middleware is not an easy job. In this paper, we figured out the issues we may encounter when designing an MMOG middleware. We discuss them in four criteria: scalability, flexibility, easy-to-use, and high performance.
In scalability, we use client-gateway-server architecture to scale up the platform.
In view of real world, we can scale up the number of clients connected by way of multiple gateways. In view of virtual world, we can scale up the number of game objects by way of spanning regions over the server cluster.
In flexibility, we reserve as few footprints as possible for the protocol. We also reserve the most resilience of game objects for developer. In addition, developer can extend the platform by designing their own plugin. There are also some flexibility concerns in designing the feature of Avatar Migration and Region Migration. We always hope the data can be migrated successfully as well as the data format is as flexible as possible. Furthermore, we use the Login Region pattern to overcome the login problem.
In easy-to-use, we simplify the underlying complexity of MMOGs by clean and thread safe API. We provide three layer of API: NetEngine library, DOIT API, plugin framework. Using DOIT API, game developer can develop a MMOG rapidly. We also suggest a easy way to send updates and receive update from an game object (by means of GameSpace and GameSpaceUtil). We also purpose a way to separate
the AI logic and game logic when designing the logic of NPC.
In high performance, we use an abstract NetEngine library, and different threading model for different demand. We deploy the event-driven NetEngine at the external interface of gateway for serving numerous clients concurrently while deploying the threaded NetEngine at the internal interface of gateway for exploiting the message aggregation feature to gain better performance. We also provided 3 experiments on our platform. We found, in some configuration, it got good performance when there are 4000 players on the same virtual world concurrently with 2 gateways and 2 servers. The bottleneck lay on the gateway component. It can hold 5000 messages per second with acceptable response time. We also can add more gateways in this platform if we need better service quality and quantity.
7.2. Future Works
In the current work, we focused on the scalable, flexible, high performance and easy-to-use issues. However, there are still plenty of issues should be taken into consideration.
Failover mechanism is absent in the current work. We hope that if a game server is done, we can recover the game states and continue to serve the clients. We may make use of the Serializable Capsule mechanism in Region Migration. We serialize the region state to the persistence store instead of another game server.
Cheating prevention is another place that has not been taken into consideration in the current work. The cheating prevention can be implemented as a gateway plugin.
We may allow the game developers to monitor and the incoming command messages and filter out the illegal message. Besides, we may provide the SSL version of
NetEngine in order to meet the requirement for high demand of online game, such as online shopping mall.
We also hope to provide a script language framework for game developers for writing game logics. There are some good script language implementation for java platform, like Jython and Groovy. We may provide a scripting framework on top of the DOIT API.
Furthermore, the easy-to-manage is also important for MMOG middlewares. The components are distributed over plenty of machines. It is desirable to have a centralize management console. In fact, we have made efforts on this feature. We adopt the JMX[19] and JMX Remote technology.
The last one is the location-free communication. In the current work, it is easy to implement the feature that the avatars talk to others around him. However, we didn’t provide the mechanism to communicate with a group located around the future world in the current work. We hope to provide this kind of location-free communication in our framework. Maybe the JMS[8] is a good choice to implement this feature.
Bibliography
[1] Zona Inc., http://www.zona.net/
[2] Butterfly.net, http://www.butterfly.net/
[3] Nathan Sheldon, Eric Girard, Seth Borg, Mark Claypool, Emmanuel Agu. The Effect of Latency on User Performance in Warcraft III. In Proceedings of ACM NetGames at Electronic Arts Headquarters Redwood City, California, USA May 22-23, 2003.
[4] M. Mauve, S. Fischer, J. Widmer. A generic proxy system for networked computer games, In Proceedings of ACM NETGAMES ’02, pp.25-28.
[5] Wish Engine, http://www.mutablerealms.com/
[6] Massively Multiplayer Middleware ,
http://www.acmqueue.com/modules.php?name=Content&pa=showpage&pid=1 15
[7] Xuan-Feng Lee, Tsun-Yu Hsiao, Shyan-Ming Yuan. A Scalable Middleware Architecture for Supporting Massive Multiplayer Virtual Worlds. In Proceeding of Symposium on Digital Life and Internet Technologies 2003.
[8] Sun Microsystems Inc. Java Message Service, http://java.sun.com/products/jms/
[9] BigWorld Technology, http://www.bigworldgames.com/
[10] K. Lee and D. Lee. A Scalable Load Balancing Scheme for Large-Scale
Multi-Server Distributed Virtual Environment Systems. In Proceedings of ACM Symposium on Virtual Reality Software and Technology (VRST2003), Osaka, Japan, 1-3 October 2003.
[11] P. Morillo, J.M. Ordu.na, M.Fern andez. An Adaptive Load Balancing
Technique for Distributed Virtual Environment Systems. Proceedings of the IASTED International Conference Parallel and Distributed Computing and Systems November 3-5, 2003 in Marina del Rey, CA, USA.
[12] Lars Aarhus, Knut Holmqvist and Martin Kirkengen. Generalized TwoTier Relevance Filtering of Computer Game Update Events. Proceedings of ACM NetGames 2002.
[13] Ashwin R. Bharambe. Mercury: A Scalable Publish/Subscribe System for Internet Games. Proceedings of ACM NetGames 2002.
[14] Sandeep Singhal, Michael Zyda. Networked Virtual Environments, Design and Implementation. Published by Addison-Wesley, 1999.
[15] Daniel Bauer. Network Infrastructure for Massively Distributed Games.
Proceedings of ACM NetGames 2002.
[16] Matt Welsh, Steven D. Gribble, Eric A. Brewer, and David Culler. A Design Framework for Highly Concurrent Systems. UC Berkeley Technical Report UCB/CSD-00-1108, Submitted for publication, April, 2000.
[17] Sun Microsystems Inc. JavaBeans Architecture,
http://java.sun.com/products/javabeans/
[18] Sun Microsystems Inc. Java Servlets API,
http://java.sun.com/products/servlet/
[19] Sun Microsystems Inc. Java Management Extension,
http://java.sun.com/products/JavaManagement/
Appendix A System Protocol
ID Name Direction Description
0x01 CTRL GÆS The control message 0x02 UPDATE GÆS The update message 0x03 AVACONN GÆS Notify that a channel connected.
0x04 AVADISC GÆS Notify that a channel disconnected
0x05 AVACLOSE SÆG Notify that a avatar channel is close by game logic
0x06 AVAMIG GÆS SÆG
Send when a avatar migrated.
0x07 GATHEL GÆC GÆS
Say hello and tell the gateway information.
0x08 SRVHEL SÆC Say hello and tell the server information
0x09 RMINIT C broascast(*1) Send when a coordinator initiate an region migration
0x0A RMREADY SÆC The destination server says he is ready to receive migration data.
0x0B RMCOMPLETE SÆC C broadcast(*1)
Send when the region migration is complete
0x0C RMFAIL SÆC C broadcast(*1)
Send when a region migration fails.
0x0D AVACONN_ACK SÆG ACK of AVACONN 0x0E AVAMIG_ACK SÆG ACK of AVAMIG
G: Gateway, S: Server, C: Coordinator
*1 A coordinator broascasts the messages to evey participents of the region migration.
0x01 CTRL Direction: GÆS Description:
The control message Format:
long avatarid short controlType short controlLength byte[] wrappedMessage
0x02 UPDATE Direction: GÆS Description:
The update message Format:
long avatarid short updateType short updateLength byte[] wrappedMessage
0x03 AVACONN Direction: GÆS Description:
Notify that a client connected.
Format:
long avatarid
0x04 AVADISC Direction: GÆS Description:
Notify that a client disconnected.
Format:
long avatarid
0x05 AVACLOSE Direction: SÆG Description:
Notify that a avatar channel is close by game logic Format:
long avatarid
0x06 AVAMIG
Direction: GÆS, SÆG Description:
Send when a avatar migrated.
Format:
long avatarid int srcRegionid int destRegionid int x
int y
short dataLength
byte[] data
0x07 GATHEL
Direction: GÆS, GÆS Description:
Say hello and tell the gateway information Format:
int gatewayid
0x08 SRVHEL Direction: SÆC Description:
Say hello and tell the server information Format:
int serverid
0x09 RMINIT
Direction: C broadcast Description:
Send hen a coordinator initiate a region migration Format:
int rmid
int regionid int srcServerid int destServerid
0x0A RMREADY Direction: SÆC, CÆS Description:
The destination server says he is ready to receive migration data. This message will be forwarded to source server by the coordinator.
Format:
int rmid int host int port
0x0B RMCOMPLETE Direction: SÆC, C broadcast Description:
Send when the region migration is complete.
Format:
int rmid
0x0C RMFAIL
Direction: SÆC, C broadcast Description:
Send when the region migration fails.
Format:
int rmid
0x0D AVACONN_ACK Direction: SÆG
Description:
ACK of AVACONN Format:
long avatarid
0x0E AVAMIG_ACK Direction: SÆG Description:
ACK of AVAMIG Format:
long avatarid