The design and implementation of the NCTUns network simulation engine

(1)

The design and implementation of the NCTUns

network simulation engine

S.Y. Wang

*

_{, C.L. Chou, C.C. Lin}

Department of Computer Science, National Chiao Tung University, 1001 Ta-Hsueh Road, Hsinchu 300, Taiwan Received 19 September 2005; received in revised form 13 May 2006; accepted 26 September 2006

Abstract

NCTUns is a network simulator running on Linux. It has several unique advantages over traditional network simula-tors. This paper presents the novel design and implementation of its simulation engine. This paper focuses on how to com-bine the kernel re-entering and discrete-event simulation methodologies to execute simulations quickly. The performance and scalability of NCTUns are also presented and discussed.

Keywords: Network simulator; Simulation methodology

1. Introduction

Network simulators implemented in software are valuable tools for researchers to develop, test, and diag-nose network protocols. Simulation is economical because it can carry out experiments without the actual hardware. Simulation is ﬂexible because it can study the performances of a system under various conditions. Simulation results are easier to analyze than experimental results because they are repeatable. In addition, in some cases involving high-speed movements (e.g., vehicle movements in Intelligent Transportation Systems), conducting simulations is much safer than conducting real experiments.

Developing a complete network simulator requires much time and efforts. A complete network simulator needs to simulate the hardware characteristics of networking devices (e.g., hub or switch), the protocol stacks employed in these devices (e.g., the bridge-learning protocol used in a switch), and the execution of application programs on these devices (for generating realistic network traffic). It also needs to provide network utility programs for configuring network topologies, specifying network parameters, monitoring traffic flows, gath-ering statistics about a simulated network, etc. Developing a high-fidelity and easy-to-use network simulator requires a large effort. Due to limited development resources, most traditional network simulators usually have the following limitations:

* _{Corresponding author. Tel.: +886 35 131 550.}

E-mail address:shieyuan@csie.nctu.edu.tw(S.Y. Wang).

(2)

• Simulation results are only abstracted version of the results produced by real hardware and software equip-ments. To constrain their complexity and development cost, most network simulators simulate real-life net-work protocol implementations with limited details, and this may lead to incorrect results. For example, OPNET’s modeler product[1]uses a simplified finite state machine model to model complex TCP protocol processing. As another example, in ns-2[2]package, it is self-documented that ‘‘there is no dynamic recei-ver’s advertised window for TCP.’’ In addition, in a real-life network, a UDP-based application program can change its destination host (i.e., change the used destination IP address for an outgoing UDP packet) at run time on a per-packet basis. However, in ns-2, this cannot be easily done because in ns-2 the traffic sink agent (an object in C++) on the destination node must be paired with the traffic source agent on the source node at the beginning of a simulation via a scripting command. As such, a traffic generator program in ns-2 cannot dynamically change its destination node at run time like a normal real-life application program does.

• Most network simulators do not support the standard UNIX POSIX application programming interface (API) system calls. As such, existing and to-be-developed real-life application programs cannot run nor-mally to generate traﬃc for a simulated network. Instead, they must be rewritten to use the internal API provided by the simulator (if there is any) and be compiled with the simulator to form a single, large, and complex program. For example, since the ns-2 network simulator itself is a user-level program, there is no way to let another user-level application program ‘‘run’’ on top of it. As such, a real-life application program cannot run normally on a network simulated by ns-2. This means that it cannot be used to gen-erate realistic traﬃc in a ns-2 simulated network. This also means that its performance under various con-ditions cannot be evaluated in a ns-2 simulated network.

To overcome these problems, the authors in[3,4] proposed a kernel re-entering simulation methodology and used it to develop the Harvard network simulator[5]. Later on, the authors improved the methodology and used it to develop the NCTUns network simulator[6]. (For brevity, we will just call it ‘‘NCTUns’’ in the rest of the paper.) Using this methodology, life protocol stacks can be directly used to generate more real-istic simulation results and real-life application programs can be directly run on a simulated network.

The kernel re-entering simulation methodology can be applied to different simulation engine designs. The Harvard network simulator used a time-stepped method to implement its simulation engine and handle the interactions between the engine and real-life application programs. However, its simulation speed is unneces-sarily low when traffic load is light. To overcome this problem, NCTUns applies the event-driven (i.e., the dis-crete event simulation methodology [7]) method to its simulation engine. With this novel design, NCTUns handles such interactions efficiently and achieves high simulation speeds when traffic load is light.

The contribution of this paper is the novel design of combining the discrete event simulation and the kernel re-entering simulation methodologies and its detailed implementation. With this design, simulations in NCTUns can be executed quickly and precisely despite the fact that there may be some other irrelevant activ-ities occurring on the system. In addition, with this design, simulation results are repeatable because the inter-actions between the simulation engine process and all involved traﬃc generator processes are precisely controlled in the kernel.

The design and implementation presented in this paper contain novelty and they are not just straightfor-ward engineering efforts. Note that although discrete event simulation is a mature field, combining it with the kernel re-entering simulation methodology is a new challenge. The challenge is due to the fact that the objects simulated in NCTUns are not contained in a single program but rather distributed in multiple indepen-dent components running concurrently on a UNIX machine. Obviously, one possible way to handle this sit-uation is to treat this problem as a parallel simulation problem and apply parallel simulation approaches[7]to these components. However, conservative parallel simulation approaches, although simple to implement, nor-mally result in very low simulation performance under tiny lookahead values (which is the case among these components). On the other hand, optimistic parallel simulation approaches, although potentially may achieve higher performance speedups on multiprocessor machines, are complicated and difficult to implement, and result in no performance gain on a uniprocessor machine, where most simulation users run their simulation cases. Since the original event-driven approach and existing parallel simulation approaches cannot solve this problem well, this paper proposes a novel, efficient, and simple to implement approach to solve this problem.

(3)

In the rest of the paper, we ﬁrst survey related work in Section2. In Section3, we brieﬂy present the kernel entering simulation methodology, giving readers enough background to understand how the kernel re-entering and discrete-event simulation methodologies can be combined. In Section4, we move on to present the detailed design and implementation of the discrete-event simulation engine of NCTUns. In Section5, we present simulation performance under various simulation conditions. In Section 6, we discuss the scalability issue. Finally, we conclude the paper in Section 7.

2. Related work

In the literature, some approaches also use a real-life TCP/IP protocol stack to generate results [8– 12,14,16,17]. However, unlike the kernel re-entering methodology, these approaches are used for emulation purposes, rather than for simulation purposes.

Dummynet [11], when it was originally proposed, also used tunnel interfaces to use the real-life TCP/IP protocol stack in the simulation machine. However, since its release in 1997, Dummynet has changed substan-tially and now is used as a real-time traﬃc shaper or a bandwidth and delay manager in the FreeBSD kernel. It is no longer used as a network simulator.

ENTRAPID[12]uses another approach to use the real-life protocol stacks. It uses the virtual machine con-cept[13]to provide multiple virtual kernels on a physical machine. Each virtual kernel is a process and sim-ulates a node in a simulated network. The system calls issued by an application program are redirected to a virtual kernel (i.e., the application program needs to be relinked so that the system calls issued by it are replaced with IPC library calls). As such, UNIX POSIX API can be provided by ENTRAPID and real-life application programs can be run in separate address space. This approach is heavy-weight in the sense that each virtual kernel process is a huge process. As such, when simulating a large network, the required memory space is likely to be huge and the simulation speed is likely to be low, because a large number of virtual kernel processes need to be context-switched frequently.

VMware[14]can also implement virtual machines. It can provide virtual x86-like hardware environments within which a number of different operating systems can be executed. With this capability, a number of vir-tual machines can be implemented on a single real machine to act as hosts or routers. They can be configured to form an emulated network and packets generated by real-life application programs run on these virtual machines can be exchanged through the emulated network. Although VMware can be used as an emulator, it is not a network simulator. As such, its results are not repeatable and a simulation case cannot be finished sooner than the simulated time in the real life. In addition, it is very heavy-weight. A virtual machine consumes much resource and runs substantially slower than the real machine – sometimes by an order of magnitude.

Emulab[15]can provide a large testbed for network experiments. Given a virtual topology, it can map the nodes in the virtual topology to some networking devices in the real life. It can integrate nodes that run emu-lations, nodes that run simuemu-lations, and live networks. It is a software platform that automatically allocates local and distributed networking resources to support a network experiment. It, however, is neither a network simulator nor an emulator by itself.

The virtual host approach[16]allows multiple virtual hosts to be instantiated on a real FreeBSD machine. It uses the jail (8) functionality provided by FreeBSD kernel to implement virtual hosts. Only one copy of the kernel is shared among multiple virtual hosts and each virtual host uses a different portion of the file system as its own file system. A virtual host has a shell and is assigned a single IP address when it starts up. All processes that are forked in a virtual host’s shell run in the virtual host’s root file system and can use the virtual host’s IP address to communicate with a real-life network. Although this approach is light-weight, virtual hosts created using this approach cannot be formed as a generic simulated network or an emulated network. This is because each virtual host can have only one network interface (as its name suggests). As such, this approach cannot be used to simulate a network in which routers containing multiple interfaces exist.

The virtual router approach[17]allows multiple virtual routers to be instantiated on a real machine, each implemented by a user-level process. Virtual routers can be conﬁgured and connected by logical channels to form an emulated network. Such logical channels are implemented using UDP tunnels. Real-life application programs can exchange their packets through a network emulated by the virtual router approach. This is because these packets are intercepted in the kernel and then redirected to virtual routers. However, a network

(4)

formed by virtual routers is not a simulated network. Instead, it is more like an emulated network. This is because the process scheduling order among multiple virtual router processes is not controlled and thus results are not repeatable.

OPNET modeler, REAL [18], ns-2, and SSFnet [19] represent the ‘‘traditional’’ network simulation approach. In this approach, the thread-supporting event scheduler, application code (not real-life application programs) that generates network traffic, utility programs that configure, monitor, or gather statistics about a simulated network, the TCP/IP protocol implementation on hosts, the IP protocol implementation on routers, and links are all compiled together to form a single user-level program. A simulator constructed using this approach cannot easily provide UNIX POSIX API for real-life application programs to run normally to gen-erate network traffic. Although some simulators may provide their own non-standard API to let real-life appli-cation programs interact with them (via IPC library calls), real-life appliappli-cation programs still need to be rewritten so that they can use the internal API, be compiled and linked with the simulator, and be concurrently executed with the simulator during simulation.

For these traditional network simulators, they can easily use the discrete-event simulation methodology to achieve high simulation speeds. This is because all objects that need to be simulated are created and contained in the single simulator program. As such, the timestamps of all simulation events generated by them can be known in advance and triggered correctly.

As for NCTUns, only one copy of the kernel (the default one) is used to simulate many virtual hosts and routers. Using the kernel re-entering simulation methodology, a virtual host or router is implemented by redi-recting their packets into the same and only one kernel. Comparing NCTUns with the VMware approach, which uses a full operating system to implement a virtual router, and the Virtual Router approach, which uses a user-level process to implement a virtual router, NCTUns is very light-weight because no operating system or process is needed to implement a virtual host or router. In addition, all real-life application programs can exchange their packets across a network simulated by NCTUns without any modification, recompilation, or relinking. More importantly, its results are repeatable because the UNIX process scheduler in the kernel has been modified to precisely control the execution order of the NCTUns simulation engine process and all forked traffic generator processes. In contrast, the VMware and Virtual Router approaches cannot generate repeatable results.

NCTUns can be easily turned into an emulator. This is done by purposely slowing down its virtual clock and synchronizing it with the clock in the real life. If in a simulation case the virtual clock runs more slowly than the real clock (e.g., a network with very high traﬃc loads), it is impossible to ‘‘slow down’’ the virtual clock to synchronize it with the real clock. This means that the CPU power of the simulation machine is not high enough to turn the simulation case into an emulation case. In the emulation mode, with the support of user-level emulation daemons, a virtual host/router in an emulated network and a real host/router in a real network can exchange their packets through the emulated network. For example, a real TCP connection can be set up between a virtual host and a real host to exchange packets between them. The detailed design, implementation, and performance evaluations of NCTUns emulation are pre-sented in [20].

Fig. 1compares the architectures used by a traditional network simulator (e.g., OPNET, ns-2, SSFnet), a virtual kernel network simulator (e.g., ENTRAPID), and a kernel re-entering network simulator (i.e., NCTUns). This case assumes that the simulated network has six nodes and on each node there is an applica-tion program running on top of it to generate traffic. In the tradiapplica-tional network simulator approach, the appli-cation programs (abbreviated as App. in the figure) are fake appliappli-cation programs. Normally, they are very simple functions responsible for transmitting packets using some specific traffic patterns. In the virtual kernel network simulator approach, application programs are real programs. However, because their system calls need to be redirected to their corresponding virtual kernel processes, they need to be modified, recompiled, and relinked. In the kernel re-entering network simulator approach, application programs are real programs and do not need to be modified, recompiled, or relinked. In the figure, the grey area represents the place where a simulated network resides in each approach. Details about the kernel re-entering methodology will be pre-sented in Section3.

To be complete,Fig. 2compares the architectures used by a VMware network emulator and a virtual router network emulator. In both approaches, application programs are real-life application programs without any

(5)

modiﬁcation. The VMware is very heavy-weight and is not likely to support many nodes either for simulation or emulation purposes. Although the virtual router approach is less heavy-weight than the VMware approach, it is still relatively heavy-weight and thus is unlikely to support a large number of nodes. Furthermore, because

Real App. Real App. Fake App. Fake App. Simulated Nodes and Their Protocol Stacks Virtual Kernel Process Modified App. Modified App. Virtual Kernel Process Modified Kernel (To support running multiple kernels concurrently)

User level Kernel level

(c) The architecture of a kernel re–entering network simulator

Simulated Layer 2– Protocol User–level Kernel–level Modified Kernel (Real-life Layer 3+ protocol stacks are directly used.)

(b) The architecture of a virtual–kernel network simulator

User level Kernel level (a) The architecture of a traditional network simulator

Simulation Engine

Fig. 1. A comparison of the architectures used by a traditional network simulator, a virtual-kernel network simulator, and a kernel re-entering network simulator. The gray area represents the place where a simulated network resides. This case assumes that the simulated network has six nodes and on each node there is an application program running on top of it to generate traﬃc.

(6)

the process scheduling order among multiple virtual router processes is not precisely controlled, results gen-erated by the virtual router approach is not repeatable, which makes it unsuitable for being a network simulator.

Going back toFig. 1, one sees that in the kernel re-entering approach, simulation events may be gener-ated by multiple real application programs, by the modiﬁed kernel (e.g., generating a TCP retransmit time-out event), and by the simulation engine program that simulates data-link layer protocols and physical-layer link characteristics (e.g., bandwidth, delay, and Bit-Error-Rate). Because these separate entities run indepen-dently, triggering their events in the correct order is more diﬃcult to achieve than in a traditional network simulator.

The differences between this paper and other NCTUns-related papers are summarized below. In[3], the concept of the kernel re-entering simulation methodology was first proposed. In[4], the simple Harvard net-work simulator was presented as a proof-of-concept of this methodology. In [6], the improved kernel re-entering methodology and new functionalities (such as the integrated GUI environment) were presented. In [21], how NCTUns can be applied to various wireless network research areas was presented. None of these papers presents how to combine the discrete event simulation and the kernel re-entering simulation methodologies. This paper is the first one that presents this novel combination and its detailed design and implementation.

3. Kernel re-entering methodology

The initial version of the kernel re-entering simulation methodology was proposed in[3,4]. Later on, it was improved in[6]. As such, in the following, we will just present the basic concept of this methodology. After

(b) The architecture of a virtual–router network emulator Real App. Real App. User – level Kernel – level Kernel 2 Kernel 1 Real App. Virtual Router Process Virtual Router Process Real App. Kernel User – level Kernel – level (a) The architecture of a VMware network emulator

Fig. 2. A comparison of the architectures used by a VMware network emulator and a virtual-router network emulator. This case assumes that the emulated network has six nodes and on each node there is an application program running on top of it to generate traﬃc.

(7)

readers have had the required background, we will use a one-hop network as an example to present how to combine the discrete-event methodology with the kernel re-entering methodology.

3.1. Tunnel network interface

Tunnel network interfaces is the key facility in the kernel re-entering methodology. A tunnel network inter-face, available on most UNIX machines, is a pseudo network interface that does not have a real physical net-work attached to it. The functions of a tunnel netnet-work interface, from the kernel’s point of view, are no diﬀerent from those of an Ethernet network interface. A network application program can send out its packets to its destination host through a tunnel network interface or receive packets from a tunnel network interface, just as if these packets were sent to or received from a normal Ethernet interface. Currently, the NCTUns installation script automatically creates 4,096 tunnel interfaces by default. Since a tunnel network interface is a software object and occupies little memory space in the kernel, this number can be further increased with-out any problem.

3.2. Simulating single-hop networks

Using tunnel network interfaces, one can easily simulate the single-hop TCP/IP network depicted in Fig. 3(a), where a TCP sender application program running on host 1 is sending its TCP packets to a TCP receiver application program running on host 2. One can set up the virtual simulated network by performing the following two operations. First, one configures the kernel routing table of the simulation machine so that tunnel network interface 1 is chosen as the outgoing interface for the TCP packets sent from host 1 to host 2 and tunnel network interface 2 is chosen for the TCP packets sent from host 2 to host 1. Second, for the two links to be simulated, one runs a simulation engine process to simulate them. For the link from host i to host j (i = 1 or 2 and j = 3 i), the simulation engine opens tunnel network interface i’s and j’s special files in /dev and then executes an endless loop until the total simulated time has elapsed. In each step of this loop, it simulates a packet’s transmission on the link from host i to host j by reading a packet from the special file

Kernel TCP sender TCP/IP stack TCP/IP stack Tunnel Interface1 Tunnel Interface 2 Link 1 Link 2 _receiverTCP Host 1 TCP_receiver Host 2 Link 1 Link 2 TCP_sender (a) (b) Read Write User– level Simulation Engine

Fig. 3. (a) A single-hop TCP/IP network to be simulated. (b) By using tunnel interfaces, only the two links need to be simulated. The complicated TCP/IP protocol stack need not be simulated. Instead, the real-life TCP/IP protocol stack in the kernel is directly used in the simulation.

(8)

of tunnel interface i, waiting the link’s propagation delay time plus the packet’s transmission time on the link (in virtual time), and then writing this packet to the special ﬁle of tunnel interface j.

While the simulation engine is running, the virtual simulated network is constructed and alive.Fig. 3(b) depicts this simulation scheme. Since replacing a real link with a simulated link happens outside the kernel, the kernels on both hosts do not know that their packets actually are exchanged on a virtual simulated net-work. The TCP sender and receiver programs, which run on top of the kernels, of course do not know the fact either. As a result, all existing real-life network application programs can run on the simulated network, all existing real-life network utility programs can work on the simulated network, and the TCP/IP network protocol stack used in the simulation is the real-life working implementation, not just an abstract or a ported version of it. Note that the kernels on the sending and receiving hosts are the same one – the kernel of the simulation machine. This is why this simulation methodology is named ‘‘kernel re-entering methodology.’’

4. Discrete-event simulation engine

From the above description, one sees that it is natural and easy to use the time-stepped simulation approach to implement the simulation engine. Actually, for this reason, the Harvard network simulator uses the time-stepped approach to implement its simulation engine.

Although the time-stepped implementation is easy, its simulation speed can be very low. The reasons are explained below. On one hand, to achieve high simulation accuracy, one would prefer to use a small time step so that the simulation engine can poll the tunnel interfaces at a very ﬁne granularity (e.g., every one nanosecond in virtual time). This is because otherwise packets entering a tunnel interface’s output queue any time between two successive polls cannot be detected immediately and processed individually. Instead, they can only be detected and processed in a batch, and this may lower the accuracy of simulation results. On the other hand, to achieve high simulation speeds, one would prefer to use a large time step so that the virtual clock can advance quickly without wasting much time on many time steps in which there is no event to trigger. Apparently, there is a trade-oﬀ between simulation speed and simulation accuracy in this time-stepped implementation.

To achieve both high simulation speed and high simulation accuracy at the same time, it is desired that the event-driven approach can be applied to the kernel re-entering simulation engine of NCTUns. However, as will be seen later, it is not easy to do so.

4.1. Virtual clock advancing and sharing

When NCTUns is ‘‘simulating’’ a real-life network and application programs (actually it directly uses them), the virtual clock of the simulated network is kept in the user-level simulation engine process, which is responsible for advancing the virtual clock based on the event-driven approach. As in the time-stepped approach, the simulation engine process keeps a list of events sorted based on their timestamps. (A heap data structure is used to eﬃciently sort events.) To safely use the event-driven approach, it is very important that all events generated by the simulation engine process itself, the kernel, and all traﬃc generator processes must be scheduled into the simulated future (that is, the timestamp of any new event inserted into the list must be at least as large as the current time of the virtual clock), otherwise, the simulation will fail.

This problem – events are generated by multiple independent entities and they must be triggered in causal order, actually is a classical problem in the simulation society, and in the past, two parallel and distributed simulation approaches (i.e., conservative and optimistic) have been proposed to solve it. If either approach is used to solve this problem, the simulation engine process, the kernel, and every application program process must maintain their own virtual clocks and the simulation times of these virtual clocks must be synchronized by using a synchronization protocol.

Although these approaches may be able to solve this problem, they are too heavy-weight and thus may not result in good performances. In addition, they need to get and modify the source code of every used applica-tion program to correctly synchronize their simulaapplica-tion times. This will force NCTUns to lose one of its unique advantages that all real-life application programs can be directly used in a simulation without any

(9)

modiﬁca-tion. For these reasons, we did not adopt the parallel and distributed simulation approaches but instead pro-posed a novel approach for NCTUns.

The simulation engine process needs to pass the current virtual time down into the kernel. This is required for many reasons. First, the timers of TCP connections used in the simulated network need to be triggered based on the virtual time rather than the real time (recall that in NCTUns, the in-kernel TCP/IP protocol stack is directly used to ‘‘simulate’’ TCP connections). Second, for those application programs launched to generate traﬃc in the simulated network, the time-related system calls issued by them must be serviced based on the virtual time rather than the real time. For example, if one launches a ping program in a simulated net-work to send out ping packets once every second, the sleep (1) system call issued by the ping program must be triggered based on the virtual time, rather than the real time. Third, the in-kernel packet logging mechanism (i.e., the Berkeley packet ﬁlter scheme used by tcpdump) needs to use timestamps based on the virtual time, rather than the real time, to log packets transferred in a simulated network.

In addition to being able to pass the current virtual time into the kernel, the simulation engine process must perform this operation in a low-cost and fine-grain way. In a simple way, the simulation engine process can pass the current virtual time into the kernel by periodically calling a user-defined system call or calling the user-defined system call whenever the virtual time changes. (For example, the simulation engine process can call the system call once every 1 ms in virtual time.) However, the cost of this approach would be too high if we want the virtual time maintained in the kernel to be as precise as that maintained in the simulation engine process. (One such demand is that the in-kernel packet logging mechanism needs a microsecond-resolution clock to generate timestamps.) To solve this problem, we let the simulation engine process use a memory-map-ping technique to map the memory location where the current virtual time is stored to a memory location in the kernel. With this technique, now at any time the virtual time in the kernel is as precise as that maintained in the simulation engine process without any system call overhead.

4.2. Event-passing channel

Because traﬃc generator processes and the kernel may generate events and these events should be inserted into the event list kept in the simulation engine process, we use the output queue of tunnel interface 0 (tun0) as the event-passing channel among these entities. (Note: NCTUns does not use tunnel interface 0 to simulate any link in a simulation.)

If a traﬃc generator process or the kernel generates an event, since the event is always generated in the ker-nel (see Section4.3), the event will be immediately enqueued into the output queue of tun0. The user-level sim-ulation engine process will immediately detect its arrival and read it out of the queue through the /dev/tun0 special ﬁle. The details about event detection is presented in Section4.4.

4.3. Event generation

Because the simulation engine process keeps the event list, it will never miss its generated local events. As such, we focus only on the events generated by traﬃc generator processes and the kernel.

4.3.1. Traﬃc generator events

If a traffic generator process, after generating and sending out a packet, can immediately tell the simulation engine process when it will generate and send out its next packet, the simulation engine process will never miss such an event. In the following, we discuss the traffic events generated by one-way UDP, two-way request/ reply UDP, and TCP traffic generators separately.

4.3.1.1. One-way UDP traffic events: Supporting one-way UDP traffic generators is easy. Normally, a one-way UDP traffic generator uses a particular traffic pattern to send out its packets. For example, the sending process of a constant-bit-rate (CBR) packet stream may send out a packet every M milliseconds, where M can be var-ied to change the traffic load (i.e., density) of the packet stream. Actually, any traffic pattern can be used and CBR is just one special case. For example, one can use an exponential distribution for the inter-packet-trans-mission times of the packet stream to generate a Possion traffic stream.

(10)

Despite the used traﬃc pattern, normally the sending process of a one-way UDP traﬃc generator can be simply implemented as follows: (written in a pseudo language)

(1) while (simulation is not done) { (2) sleep (waitsometime);

(3) sendto (destination, packet); (4) }

The statement in line 3 is the sendto() socket system call, which sends out a packet onto the network. The statement in line 2 is the sleep() system call, which asks the UNIX kernel to put the calling process into the sleep state for the speciﬁed period of time. (The usleep() system call can also be used, which can provide a microsecond resolution.) The length of the sleeping time (i.e., the value of the waitsometime variable) determines the inter-packet-transmission time. It can be a ﬁxed value or drawn from a distribution.

To make the above one-way UDP traﬃc generator correctly work in a simulated network, NCTUns needs to perform two tasks. First, the kernel needs to service the sleep(waitsometime) system call based on the virtual time, rather than the real time. Second, the intention that after sleeping waitsometime seconds the calling pro-cess will continue to send out a packet should be represented as a timer event and immediately inserted into the event list kept in the simulation engine process.

To perform both tasks, we modiﬁed the sleep() system call so that when the sleep(waitsometime) sys-tem call is called, a KERNEL-TIMER-SETUP event recording the wake-up time is created and inserted to the event-passing channel. The simulation engine process will immediately detect the arrival of such an event, read it out, and then insert it into its event list. The event detection procedure is detailed in Section 4.4.

4.3.1.2. Two-way request/reply UDP traffic events: Supporting two-way request/reply UDP traffic generators can also be easily done. The receiving process of a two-way request/reply traffic generator can be simply imple-mented as follows:

(1) while (simulation is not done) {

(2) recvfrom (& requestPacketBuffer, & source); (3) sendto(source, replyPacket);

(4) }

In line 2, the receiving process calls a recvfrom() socket system call to wait for a request packet to arrive. The recvfrom() is a blocking system call, which means that it will block (i.e., the calling process will be put into the sleep state) in the kernel if no request packet arrives. If a request packet arrives, its content will be copied to the provided packet buﬀer (i.e., requestPacketBuﬀer) and the IP address and port number of the sending process will be copied to the provided address variable (i.e., source). In line 3, the receiving process calls a sendto() socket system call to send back its reply packet, using the IP address and port number stored in the source variable. After sending back the reply packet, the receiving process will again get blocked on the recvfrom() system call, waiting for the next request packet to arrive.

The following summarizes the involved processing when the kernel receives a packet from a network inter-face. After a packet is received, it will be processed by the TCP/IP protocol stack and finally inserted into the socket receive buffer used by the receiving process. The sleeping receiving process (which is now blocked on the recvfrom() system call) will then be waked up by the kernel and its state be restored back to the ready state. When the receiving process is scheduled again (i.e., given the CPU), it will restart from the middle of the recv-from() system call where it was blocked. The recvrecv-from() system call will copy the packet from the socket receive buffer to the packet buffer provided by the receiving process and then return from the kernel to the receiving process. When getting the CPU again, the receiving process will then call the sendto() system call to send out a reply packet. This reply packet will be inserted into the output queue of some tunnel interface used by the receiving node.

(11)

Section3.2illustrates that when a request packet arrives at the receiving node, the simulation engine pro-cess will write it into the kernel to deliver it to the receiving propro-cess. The receiving propro-cess may or may not send back a reply packet. However, because the event-driven approach requires that a generated event (here, it is the generated reply packet) be immediately detected and inserted into the event list, we let the simulation engine process immediately check all involved tunnel interfaces after it writes a packet into the kernel. Here, a tunnel interface is involved in a packet-write operation if it is used by the node on which the receiving process is run. If any packet is found in the output queues of these involved tunnel interfaces, the simulation engine process will immediately get it, make it an event, insert it into the event list, and start simulating sending it back on the reverse link. Since the simulation engine process does not advance its virtual clock during this period of time, this event is detected immediately after its creation in virtual time.

In the above description, we implicitly assume that after the simulation engine process writes the request packet into the kernel and before it checks all involved tunnel interfaces, the receiving process has had a chance to gain the CPU to generate a reply packet. However, this property is not guaranteed without explicitly controlling the scheduling order between the simulation engine process and the traﬃc generator processes. We present the details of this scheduling design in Section4.7.

Of course, after receiving a packet, not every receiving process will generate and send back a reply packet. For example, in the pseudo code, if the sendto() system call in line 3 is removed, the receiving process will become a normal one-way receiving process. Therefore, sometimes checking involved tunnel interfaces does not necessarily find any packet to be sent. However, this does not affect the correctness of simulation results. It just affects simulation performance insignificantly.

The above design not only applies to the UDP protocol, but also applies to the ICMP protocol. For exam-ple, it can support the ping request/reply traﬃc generator equally well, which uses ICMP’s echo request/reply commands. Since the ICMP protocol stack resides in the kernel, after an ICMP echo request packet is received by the kernel, the ICMP echo reply packet will be directly generated by the kernel, not by any user-level receiv-ing process. Although the ICMP request/reply processreceiv-ing is diﬀerent from the UDP request/reply processreceiv-ing, the same event-driven design can handle both cases correctly.

4.3.1.3. TCP traffic events: Supporting TCP traffic generators is more difficult than supporting UDP traffic generators. This is because, unlike in the UDP traffic generator case, when to send out TCP packets onto a network actually is triggered by the TCP protocol module in the kernel (due to TCP congestion control), rather than directly by a user-level sending process. In addition, the TCP protocol module uses a self-clocking mechanism (i.e., send out new data packets when receiving ACK packets) to automatically send out its pack-ets. As such, unlike in the UDP traffic generator case, sending a TCP data packet is not preceded by a sleep() system call. The simulation engine process therefore cannot easily know the TCP module’s intention to send out its next packet. In the following, we use a greedy TCP transfer to explain how the event-driven design can be used. It is assumed that the sending and receiving nodes are directly connected by a link.

4.3.1.3.1. Connection setup phase: Here we focus on the TCP connection setup phase, which can be sup-ported by using the same design for supporting request/reply traffic generators. On the receiving node, a receiving process (e.g., a web server) will call a blocking accept() socket system call to wait for a connection request packet (i.e., the first SYN packet of the TCP 3-way handshaking connection setup procedure) to arrive. On the sending node, a sending process (e.g., a web client) will call a connect() socket system call to send out the connection request packet (i.e., the first SYN packet). The SYN packet will be enqueued into the output queue of a tunnel interface (which simulates the link in the forward direction), and the simulation engine process will immediately detect it, make it an event, and then insert it into the event list. The event then will be immediately triggered and processed, which starts simulating sending this packet onto the link. After the packet has arrived at the other end of the link, the simulation engine process will write it into the kernel, and the SYN packet will be received by the TCP module in the receiving node.

The TCP module in the receiving node will immediately generate the second connection setup packet (i.e., the SYN + ACK packet) and send it back. This packet will then be enqueued into the output queue of the tunnel interface that simulates the link in the reverse direction. As in the two-way request/reply traﬃc gener-ator case, the reply packet will be immediately detected by the simulation engine process and read out of the kernel, and the simulation engine process will start simulating sending this packet onto the link in the reverse

(12)

direction. When the packet arrives at the other end of the link, the simulation engine process will write it into the kernel. The TCP module in the sending node will then generate the third connection setup packet (i.e., the ACK packet). Again as in the two-way request/reply case, this packet will be detected and read by the sim-ulation engine process immediately, and the simsim-ulation engine process will start simulating sending it onto the link in the forward direction. When the packet arrives at the receiving node, the TCP connection’s 3-way handshaking connection setup procedure is ﬁnished. At this time, both the sending and receiving pro-cesses will return from their connect() and accept() system calls, respectively.

4.3.1.3.2. Data transfer phase: Here we focus on the TCP data transfer phase. Before we present the event-driven design, we first summarize the behavior of the sending and receiving processes in normal uses. After the connection is set up, the sending process will continuously call the send() socket system call to write its data into its socket send buffer. If the socket send buffer is not full, the send() system call will immediately return, allowing the sending process to write more data into the socket. Otherwise, the send() system call will get blocked and the sending process will be put into the sleep state. When the socket send buffer becomes non-full (because some data have been sent and their corresponding ACK packets have come back), the kernel will wake up the sending process, enabling it to write more data into the socket. The pseudo code of the sending process is shown below.

(1) connect(destination);

(2) while (simulation is not done) { (3) send (destination, moreNewData); (4) }

On the other hand, the receiving process will continuously call the receive() socket system call to try to read more data out of its socket receive buﬀer. Since the receive() system call is a blocking system call, if there is no data in the socket receive buﬀer to be read out, the receiving process will be put into the sleep state. The kernel will wake it up when some new data packets have arrived. The pseudo code of the receiving process is shown below.

(1) accept(& source);

(2) while (simulation is not done) { (3) receive(source,& packetBuffer); (4) }

In the above description, we just describe when data is written into/read out of a socket buﬀer and do not describe when packets are sent onto the network. This is because the TCP protocol implements error, ﬂow, and congestion controls. It is the TCP protocol that determines when to send out a packet, not the sending process. In contrast, since the UDP protocol implements no such controls, it is the sending process that deter-mines when to send out a packet, not the UDP protocol module. In the following, we will present how to use the event-driven approach to support asynchronous TCP packet transfers.

We ﬁrst focus on the TCP module in the receiving node. When a data packet arrives at the receiving node, normally the TCP module will send back an ACK packet immediately. In addition, because TCP uses a cumu-lative ACK scheme, when a data packet is lost, the TCP module in the receiving node will send back a dupli-cate ACK packet whenever an out-of-order data packet is received. Both of these scenarios can be supported by using the design for supporting the request/reply traﬃc generators. If the TCP module uses the delay-ACK option, which may delay an ACK packet transmission by up to 200 ms for trying to piggyback some data, the transmission of the ACK packet will be triggered by the expiration of the TCP’s fast timer. The design for supporting this scenario is presented in Section4.3.2.

Then, we focus on the TCP module in the sending node. Although we said earlier that the times when the TCP module sends out a packet are independent of the times when the TCP sending process calls the send() system call, in some cases the TCP module’s output function (i.e., tcp_output() in BSD implementation) still needs to be directly triggered by the TCP sending process calling the send() system call. This happens when the send() system call ﬁnds that the TCP connection currently has no data packets outstanding in the network and

(13)

its retransmit timer is not active (e.g., the TCP connection just finishes its connection setup phase or the TCP connection currently is idle). In such a case, since no ACK packet is expected to come back to trigger out more data packets (i.e., self-clocking), the send() system call must send out the data by itself and this is done by directly calling the TCP module’s output function (tcp_output()). On the other hand, if the send() system call finds that either there are outstanding data packets in the network or the TCP retransmit timer is already active, it will just insert the data into the socket send buffer and simply return. This is because the send() sys-tem call knows that its data will later be sent out by tcp_output(), which will either be triggered by a returning ACK packet or the expiration of the retransmit timer.

Having presented the TCP output procedure, we now present how the event-driven approach is used to support asynchronous TCP data transfers. First, it is clear that data packets generated and sent due to the arrival of a returning ACK packet can be supported by the same design for supporting request/reply traffic generators. Second, the data packet generated and sent due to directly calling tcp_output() by the send() sys-tem call can be immediately detected by the simulation engine process. This is because if the data packet is triggered out due to the arrival of the last packet of the 3-way handshaking connection setup procedure (like what a web server does), this first data packet can be supported by the same design for supporting request/ reply traffic generators. If the data packet is not the first data packet but a packet generated when the TCP sending process becomes active again after an idle period of time, then before the sending process calls the send() system call, it must have called a sleep() system call. Apparently, this can be supported using the same design for supporting one-way UDP traffic generators presented in Section4.3.1.1. The last reason why a data packet would be generated and sent is due to the expiration of the TCP retransmit timer. The design for sup-porting this case is presented in Section 4.3.2.

4.3.2. Kernel events

Here we present the event-driven design for supporting the events generated inside the kernel. Kernel events come from two main categories. The ﬁrst category is TCP timer events. A TCP connection has several timers. For example, it has retransmit, persist, keepalive, delay-ACK, and 2MSL timers. Except for the 2MSL timer, which deallocates all the resources used by an already-closed TCP connection, when a TCP timer expires, a packet will be generated and sent onto the network for some purposes. The second category is the timer events generated by traﬃc generator processes calling time-related system calls such as sleep() and alarm().

Although these timer events are generated due to diﬀerent reasons and from diﬀerent control paths, starting from FreeBSD 3.0 (right now the latest FreeBSD version is 6.1), they all map to callout requests and are trig-gered by the same callout mechanism in the kernel. (The Linux kernel adopts a similar design.) In each invo-cation of the clock interrupt service routine, the real time clock maintained in the kernel is advanced by one tick (10 ms or 1 ms in real time, depending on whether the kernel HZ option is set to 100 or 1000 when the kernel was compiled). The clock interrupt service routine also checks whether the current real time is the same as the triggering time of some callout request events. If any such event is found, it will call the event’s asso-ciated function (e.g., the TCP retransmit function).

To use the event-driven approach to quickly trigger these timer events in virtual time, upon creation, these timer events are immediately passed to the simulation engine process through the event-passing channel. How the simulation engine process can immediately detect these events is presented in Section 4.4.

To let the simulation machine still work properly while NCTUns is running, only those events generated by traffic generator processes and their TCP connections are triggered based on the virtual time. All other events are still triggered based on the real time. In order for the kernel to distinguish between traffic generator pro-cesses and all other application propro-cesses, when a traffic generator process is forked by the simulation engine process, it is automatically registered with the kernel. In addition, later when it creates a TCP connection, the TCP connection is also automatically registered with the kernel. With this registration information, the kernel can thus correctly trigger events based on different clocks.

4.4. Event detection

Upon creation, if not created by the simulation engine process, an event needs to be passed to the simula-tion engine process and inserted into the event list immediately. For the event-driven approach to work

(14)

correctly, ‘‘immediately’’ here means that it takes no time in virtual time, that is, the virtual clock should be frozen between the time when an event is created and the time when it is inserted into the event list. In previous sections, we have presented various scenarios in which events may be generated and inserted into the event-passing channel. Now we present how the simulation engine process can immediately detect these events. That is, how does it know that an event has just been inserted into the event-passing channel?

In the NCTUns design, there is no notification mechanism for notifying the simulation engine process that an event was just created and inserted into the event-passing channel. This is because these events already serve notification purposes. One cannot recursively use another kind of notification events to notify the sim-ulation engine process of the arrivals of these ‘‘notification’’ events. Instead, the simsim-ulation engine process must know when an event may just have been created and inserted into the event-passing channel by itself. Indeed, the simulation engine process has the capability to perform this task. This capability comes from the fact that an event can be created only after the simulation engine process has performed one of three dif-ferent operations. Therefore, the simulation engine process can actively check the event-passing channel every time when it has performed any of such operations. In the following, we present these different operations. 4.4.1. Forking a traffic generator process

After a traffic generator process is forked by the simulation engine process, it will begin to do some work. For the one-way UDP traffic generator case in Section4.3.1.1, it will call the sleep() system call and cause a KERNEL-TIMER-SETUP event to be generated and inserted into the event-passing channel. For the request/reply traffic generator case in Section4.3.1.2, it will get blocked in the kernel when executing the recv-from() system call and generate no event. For the TCP traffic generator case, the sending process will generate a PACKET-GEN event and cause it to be inserted into the event-passing channel when it calls the connect() system call. On the other hand, the receiving process will get blocked in the kernel and generate no event when executing the accept() system call.

From the above description, we see that after forking a traﬃc generator process, some events may be gen-erated and inserted into the event-passing channel. The simulation engine process thus needs to check the event-passing channel after forking a traﬃc generator process.

4.4.2. Writing a packet into a tunnel interface’s special ﬁle

In Section4.3, in the request/reply and TCP traffic generator cases, we see that some events may be gen-erated after the simulation engine process writes a packet into a tunnel interface’s special file (e.g., /dev/tun3). For example, when an ACK packet arrives at the sending node, the TCP module in the sending node may (1) trigger out more data packets, (2) cancel the current retransmit timer, and (3) set up the retransmit timer again using a fresher timeout value. These operations will generate PACKET-GEN, KERNEL-TIMER-CANCEL, and KERNEL-TIMER-SETUP events, respectively, and cause them to be inserted into the event-passing channel. The simulation engine process thus needs to check the event-passing channel after writing a packet into a tunnel interface’s special file.

4.4.3. Triggering a callout request

After the simulation engine process triggers a KERNEL-TIMER-SETUP event (presented in Section4.5), it also needs to check the event-passing channel because some events may have been generated. For example, in the one-way UDP traﬃc generator case in Section4.3.1.1, after a KERNEL-TIMER-SETUP event is trig-gered by the simulation engine process, the traﬃc generator process will (1) return from the sleep() system call, (2) call the sendto() system call, and then (3) call the sleep() system call again, which makes it blocked again in the kernel. The above step (2) and (3) will generate a PACKET-GEN and KERNEL-TIMER-SETUP event, respectively, and cause them to be inserted into the event-passing channel. Therefore, the simulation engine process also needs to check the event-passing channel after triggering a callout request event.

4.5. Event triggering

In the event-driven approach, the simulation engine process always repeats the following cycle: (1) jump to the triggering time of the event with the smallest timestamp, and (2) trigger the event by performing the

(15)

func-tion associated with the event. The events that may be stored in the event list can be classiﬁed into four cat-egories. In the following, we present them separately.

The events in the ﬁrst category are PACKET-GEN events. A PACKET-GEN event indicates that a packet has just been generated and inserted into the output queue of some tunnel interface (e.g., tun4). When the sim-ulation engine process triggers such an event, it will read a packet out of the output queue of the speciﬁed tunnel interface and start simulating sending it onto a link. These events must be triggered immediately after they are created and detected.

The events in the second category are KERNEL-TIMER-SETUP events. A KERNEL-TIMER-SETUP event indicates that a kernel timer should be set up and expire at the specified time. When triggering such an event, the simulation engine process calls a user-defined system call, which is provided by NCTUns. The system call will find the corresponding timer event in the kernel’s callout request queue, dequeue it, and call the function associated with this timer event. For example, if a KERNEL-TIMER-SETUP event was created due to a traffic generator process calling a sleep() system call, triggering this event will simply change the state of the calling process from SLEEP to READY and make the sleep() system call return from the kernel. On the other hand, if such an event was created due to setting up a TCP retransmit timer, triggering such an event will call the TCP module’s retransmit function.

The events in the third category are KERNEL-TIMER-CANCEL events. A KERNEL-TIMER-SETUP event, after being detected and inserted into the event list, may need to be canceled before it is triggered. For example, a KERNEL-TIMER-SETUP event created due to the TCP module setting up a TCP retransmit timer will likely need to be canceled before its triggering. This is because (1) the TCP module will set up a TCP connection’s retransmit timer whenever it sends out a data packet and detects that the retransmit timer is not active, and (2) it will cancel the retransmit timer as soon as the corresponding ACK packet comes back. Trig-gering a TIMER-CANCEL event can be done by simply ﬁnding the corresponding KERNEL-TIMER-SETUP event and remove it from the event list.

The events in the fourth category are all other events generated by the simulation engine process itself. For example, a MAC802.3 module may generate a timer event whose expiration will cause a packet to be sent after an exponential backoﬀ period.

4.6. Requirements for traﬃc generator processes

For the event-driven approach to work correctly, NCTUns requires that if a traﬃc generator process would send out a packet in the future, the intention must be passed to the simulation engine in advance (e.g., by pre-ceding a sendto() system call by a sleep() system call). Otherwise, the traﬃc generator process’s sending out a packet must be an immediate response to the arrival of an packet (e.g., sending back an ACK packet in response to the arrival of a data packet). As we have presented previously, in the current UNIX network sub-system design, these requirements are automatically met.

Another requirement is that a traffic generator process cannot be a CPU-bound busy-looping process. That is, it cannot keep generating and dumping packets onto the network without sometimes calling a sleep() sys-tem call to release the CPU. The reason is clear as the simulation engine process needs to gain CPU cycles to advance its virtual clock. If a traffic generator process would keep using the CPU without releasing it, the sim-ulation engine process will have no chance to advance the virtual clock and thus the whole simsim-ulation cannot proceed. Therefore, NCTUns requires that when the simulation engine process releases its CPU control to a traffic generator process, the traffic generator process, after executing some statements of its program, must get blocked again (either by explicitly calling sleep() or automatically getting blocked due to no packet to receive). This requirement ensures that the simulation engine process can acquire the CPU control most of the time to advance the simulation.

Fortunately, almost all normal traffic generators meet this requirement automatically. For the greedy TCP connection case, because TCP implements congestion control (for fairly sharing the available bandwidth in a network) and flow control (for not to overflow the receiving process’s socket receive buffer), the TCP sending process cannot keep using the full CPU speed to continuously generate and dump TCP packets onto the net-work. Instead, it can only dump its congestion window size worth of data per its connection’s round-trip-time (RTT) in virtual time. As such, the TCP sending process actually is put into the sleep state most of the time.

(16)

The TCP receiving process also is put into the sleep state most of the time because new data cannot contin-uously arrive in each CPU cycle.

For the greedy UDP traﬃc generator case, this could be a problem if the sending process is implemented in the following way:

(1) while (simulation is not done) { (2) sendto (destination, packet); (3) }

Fortunately, most existing greedy UDP traﬃc generator sending processes (e.g., the ttcp or the netperf per-formance benchmarking program that can be easily fetched on the Internet) are implemented in the following way:

(1) while (simulation is not done) {

(2) result = sendto(destination, packet); (3) if (result == noBuffer)

(4) sleep(sometime);

(5) }

In line 2, the sending process gets the result of sending a UDP packet. In lines 3 and 4, if the result indicates that the UDP packet is dropped on the local host due to a full queue (the UNIX UDP/IP pro-tocol stack can return this no-buffer-available information), the sending process puts itself into the sleep state for a while (the period is usually set to a value that is large but can still keep the output queue non-empty at all time). The reason for doing this is evident. If the sending process does not put itself to sleep for sometime, it will keep generating and sending packets and most of these packets will be dropped on the local host without having a chance to enter the network. For example, on a high-speed machine equipped with a 3.0 GHZ Pentium processor, if a greedy UDP sending process keeps using the full CPU speed to continuously send packets onto a network through a 10 Mbps Ethernet card, more than 90% of its packets will be dropped on the local host due to buffer overflow in the Ethernet device driver’s output queue.

Since the bad implementation causes problems for the sending process’s data error recovery, provides no extra beneﬁt for the sending process, and only unnecessarily wastes CPU cycles, all greedy UDP traﬃc gen-erators provided by NCTUns use the good implementation.

4.7. Explicit process scheduling

Correctly scheduling the simulation engine process and all forked traﬃc generator processes is very impor-tant for the event-driven approach to function correctly. Since the default UNIX process scheduler uses a dynamic priority mechanism and thus cannot guarantee a desired scheduling order, we have to solve this problem.

As presented in Section4.6, when the simulation engine process wants to release the CPU to a traffic gen-erator, we should let the intended traffic generator process really gain the CPU control and execute some of its program statements until it gets blocked again. Also, when the traffic generator process gets blocked again, we should let the simulation engine process regain the CPU. Ensuring this correct operation sequence, however, requires us to fully control UNIX process scheduling.

For example, for the one-way UDP traffic generator case presented in Section4.3.1.1, after the simulation engine process triggers (i.e., calls the user-defined system call presented in Section4.5) the KERNEL-TIMER-SETUP event created due to the traffic generator process calling the sleep() system call, the simulation engine process will expect that the traffic generator process would have returned from its sleep() system call, called the sendto() system call to send out a packet, and called the sleep() system call again and now blocked. This requires that the UNIX process scheduler explicitly switch the CPU control from the simulation engine pro-cess to the traffic generator propro-cess, and then from the traffic generator propro-cess back to the simulation engine

(17)

process. We found that this requirement can be automatically met due to the default UNIX process scheduling design without modifying the default UNIX process scheduler.

In the default UNIX process scheduling design, a user-level process has two priorities – one for its execu-tion in the user mode and the other is for its sleeping in the kernel mode. When a process gets blocked in the kernel, its kernel-mode priority will temporarily be assigned a very high priority, which is higher than any user-mode priority. As such, when it is waked up in the kernel, it can immediately regain the CPU, finish its system call, and return from the system call. (This design is to avoid holding too many valuable resources in the kernel, which may cause resource deadlock situations.) Due to this design, when the simulation engine process performs its operation (e.g., triggering a KERNEL-TIMER-SETUP event) and causes a sleeping traf-fic generator process to be waked up, the traffic generator process (which is still in the kernel mode) will auto-matically take the CPU control from the simulation engine process (which is in the user mode). Because in NCTUns, a traffic generator process will get blocked again after executing some program statements, the CPU control will again automatically be transferred to the simulation engine process, which allows the sim-ulation to proceed.

5. Simulation performance

In the following, we present the performances of the event-driven version of NCTUns simulation engine under various traﬃc conditions. We also compare the performances of the event-driven version of NCTUns (simply named NCTUns), the time-stepped version of NCTUns (named NCTUnsTS), and ns-2. For NCTUnsTS, it polls all tunnel interfaces used in a simulation case every 1 ms in virtual time to detect packet events.

The used machine for the performance testing is an IBM T30 notebook computer equipped with a 1.6 GHZ Pentium-M processor and 256 MB RAM. The operating system used is Fedora Linux 1.0, which is a non-com-mercial descendent of Red-Hat 9.0 distribution with Linux kernel version 2.4.22.

5.1. Self-evaluation

In this test suite, the network topology is a single-hop network in which a sending host and a receiving host are connected together by a link. The bandwidth of the link is set to 1000 Mbps and the delay is set to 1 ms, in both directions. The traﬃc generated is a one-way constant-bit-rate (CBR) UDP packet stream. Each UDP packet’s payload size is set to 1400 bytes.

We varied the packet interval time of the CBR packet stream to see how the simulator’s speed will change when it needs to process more events in each simulated second. The packet interval time is the time interval between two successive packet transmissions. The tested intervals are 0.00001, 0.00005, 0.00025, 0.00125, 0.00625, 0.03125, 0.15625, 0.78125, and 3.90625 s, respectively. (Note: The interval is enlarged by 5 times grad-ually in the tests.)

The performance metric reported is the ratio of the simulated seconds to the seconds required for finishing the simulation. In all of our tests, the simulated seconds is set to 3000 s. A simulation case with a higher ratio means that it can be finished more quickly than a case with a lower ratio. A simulation case with a ratio of 1 means that it needs the same amount of time in real time as the simulated time to finish simulating the case. Table 1lists the test results of the relationship between speedup ratio and CBR packet interval time. The CBR bandwidth corresponding to each CBR time interval and the elapsed time for finishing the simulation case are also listed. We see that due to the discrete-event simulation engine design, a simulation case can be finished quickly if it does not have many events (i.e., traffic) to process per simulated second. Besides, when the network traffic load is less than 232 Mbps, a simulation case can be finished sooner than real time on an IBM T30 notebook computer.

5.2. Comparison with ns-2 and NCTUnsTS

Here we compare the performances of NCTUns with those of ns-2 and NCTUnsTS. The performance met-rics used are (1) required memory space, (2) required time, and (3) required disk space for storing packet trace

(18)

log files. The required memory is the maximum size of the simulation engine process during a simulation run. This information can be returned by the ‘‘top’’ or ‘‘vmstat’’ command on UNIX. We first compare their per-formances on a fixed network. Then we compare their perper-formances on a wireless ad hoc network.

5.2.1. Fixed network

The configuration of the tested fixed network is depicted in Fig. 4. In this configuration, there are five source nodes, one destination node, and a bottleneck router node. The bandwidth and delay of all links are set to 100 Mbps and 1 ms, respectively. The maximum packet queue length of the FIFO queue in the bot-tleneck router is set to 300 packets. We conducted two tests. In the first test, there is a CBR UDP flow between each pair of a source and the destination node. That is, in total five UDP flows will compete for the bandwidth of the bottleneck link. The length of each UDP packet is 1000 bytes and their packet interval time is set to 0.00005 s. As such, the load of each UDP flow is 160 Mbps. In the second test, each CBR UDP flow is replaced with a greedy TCP flow and in total five greedy TCP flows compete for the bandwidth of the bottle-neck link. The length of each TCP packet is 1500 bytes, which is Ethernet’s MTU. In both tests, the time to be simulated for each simulation case is 100 s.

Table 2shows the performance comparisons between NCTUns, ns-2, and NCTUnsTS for the CBR UDP traﬃc case. In contrast,Table 3shows the performance comparisons between NCTUns, ns-2, and NCTUnsTS for the greedy TCP traﬃc case. Here we just show the comparison results. These results are discussed later in Section5.3.

5.2.2. Wireless ad hoc network

The conﬁguration of the tested wireless ad hoc network is depicted inFig. 5. In this conﬁguration, there are in total 100 wireless ad hoc nodes deployed as a 10· 10 grid. Each node is equipped with a 11 Mbps IEEE 802.11(b) wireless interface operating in the ad hoc mode. The transmission and interference ranges of such a wireless interface is set to 250 and 550 m, respectively, which are the default settings used in ns-2. The

hor-Src Src Src Src Src Dst

Fig. 4. The multi-source-single-destination ﬁxed network topology used to compare the performances of NCTUns, ns-2, and NCTUnsTS. Table 1

The simulation performance under various CBR UDP traﬃc load

CBR time interval (s) CBR bandwidth Elapsed time (s) for simulating 3000 s Ratio (higher is better)

0.00001 1 Gbps 15243 0.19 0.00005 232.96 Mbps 3135 0.96 0.00025 46.59 Mbps 621 4.83 0.00125 9.32 Mbps 124 24.19 0.00625 1.86 Mbps 25 120.00 0.03125 372.73 Kbps 6 500.00 0.15625 74.55 Kbps 2 1500.00 0.78125 14.91 Kbps 1.15 2608.69 3.90625 2.981 Kbps 0.81 3750.00

(19)

izontal and vertical distance between two neighboring nodes is set to 240 m so that a node can only commu-nicate with its direct neighbors in the horizontal and vertical directions. As such, if a node on the left side of the grid would like to send a packet to another node on the right side of the grid, the packet needs to traverse 9 Table 2

Performance comparisons between (A) NCTUns, (B) ns-2, and (C) NCTUnsTS

Metric (A) (B) (C) (A)/(B) (A)/(C)

With trace ﬁle

Memory usage (KB) 16288 13612 16288 1.196 1

Required time (s) 700 635 5875 1.102 0.119

Trace ﬁle size (MB) 600 7066 600 0.085 1

Without trace ﬁle

Memory usage (KB) 12504 13427 12504 0.931 1

Required time (s) 372 51 5464 7.294 0.068

The tested traﬃc is CBR UDP traﬃc.

Table 3

Performance comparisons between (A) NCTUns, (B) ns-2, and (C) NCTUnsTS