This chapter underlines the overview of networking device testing, as well as the architecture of OpenFlow network. Finally, it discusses the related works.
2.1 Networking Device Testing
We categorize networking device testing by testing traffic into four types, including artificial traffic replay testing, live testing, real traffic replay testing and real-time capture and replay testing. Figure 1 presents the architectures of the networking device testing of the four types.
Figure 1: Testing architecture.
Figure 1 (a) presents the artificial traffic replay testing. We deploy the DUT and the traffic generator in a closed testbed, and the traffic generator will send traffic to the DUT. The traffic is produced by network debugging tools like SmartBits, Codenomicon, etc. Figure 1 (b) presents the live testing. This testing deploys DUT in a live environment. To handle network interrupt when the DUT fails, a monitor is used to check the DUT and a bypass switch is used
to allow traffic bypass the DUT when the DUT failure. Live testing is in the normal mode when the DUT works correctly. In this mode, the traffic passes through the bypass switch as if DUT connects to live network directly. When the monitor detects a failure, it will switch to the bypass mode. In this mode, the ports connected to DUT will close and the traffic will bypass the DUT. In Figure 1 (c) presents the real traffic replay testing, which is in a closed testbed. This testing replays traffic captured from a live environment by replay tools like TCPreplay. Figure 1 (d) presents the real-time capture and replay testing. This testing has two modes: live mode and replay mode. The live mode is similar to the live mode in live testing, but the difference is that the machine not only checks the DUT but also buffers live traffic.
When the failure is detected, it will turn to the replay mode. Like live testing, the bypass switch will allow traffic bypass the DUT. At the same time, the replayer can replay the traces buffered earlier to reproduce defects for debugging.
Table 1: Comparisons of testing. Traffic source Artificial traffic Live traffic Captured trace Live traffic/captured
trace
Network service quality Yes No Yes No
Traffic complexity Low High Middle Middle - high
Defect reproduction Full completed No Partial completed Most completed
Test case customization High Low Middle Low
We compare the four types of testing in Table 1. In spite of the simplicity of traffic, the advantages of artificial traffic replay testing are the ability of reproducing all defects and good test case customization. The customization of test cases is high because most traffic generators can configure the property of generated traffic. On the contrary, live testing keeps the reality of traffic, but it sacrifices service quality for unexpected network interrupts by DUTs. Moreover, it cannot reproduce the event and has bad test case customization either.
Real traffic replay testing is the compromise between the first two types of testing. It uses captured real traffic to improve the complexity and keep the ability of defect reproduction.
But the ability of defect reproduction is worse than live testing because of the limitation of replay tools and replay scenarios. It has good service quality as a closed testbed is used, and the customization is better than live testing because we can categorize the collected traces into several groups for testing. Real-time capture and replay testing has better traffic complexity than real traffic replay testing because it deploys the DUT online to encounter some un-reproducible defects, and it has better defect reproduction because more similar replay
scenario than real traffic replay testing. But the tradeoff is the network interrupt when the mode switching. Because the test cases depend on the live traffic, its customization is as low as live testing.
2.2 OpenFlow Network
OpenFlow is a protocol which provides access to the data plane of networking devices. It separates the control plane and the data plane, so that the forwarding path of networking devices can be decided by a remote controller. Administrators can change the network topology from a software controller, significantly enhancing the flexibility of network traffic management.
An OpenFlow network consists of two components: OF switch and OF controller. An OF switch transmits data packets according to a flow table and interacts with the OF controller by a secure channel. When a packet arrives, the OF switch will check its flow table first. If the packet does not match any rule in the flow table, it will be sent to the controller through the secure channel. The OF controller will make a decision for this packet and add a rule into the flow table. When the next packet that belongs to the same flow arrives, the OF switch can handle it by this rule. There are many powerful OF controller implementations like NOX/POX [18], Beacon [19], Floodlight [20], etc. By programming the controller, administrators can control traffic as they want.
2.3 Related Works
Some popular traffic capture and traffic replay tools are discussed in this subsection and their comparisons are summarized in Table 2.
Table 2: Comparisons of capture and replay tools.
Capture Replay Feature
Tcpdump [4] Low volume traffic N/A Basic traffic capture Time Machine [5] High volume traffic N/A Long connection cutoff
SocketReplay [11] High volume traffic Stateful, 2 ports Long connection cutoff, payload cutoff, socket connection TCPreplay [6] N/A Stateless, 2 ports Divide traffic to server & client Tomahawk [7] N/A Stateless, 2 ports Traffic retransmission
Monkey [8] Low volume traffic Stateful, 2 ports Delay simulation,
TCPopera [9] N/A Stateful, 2 ports TCP state emulation
Volume Control Replay [10] N/A Stateful, 2 ports Replay traffic volume control
There are several studies related to capture losses and storage space saving. They are all developed from base traffic capture – Tcpdump[4], which is useful with low volume traffic but is inefficient with high-speed traffic because of the frequent system interrupts. To reduce capture losses, many studies use filtering. The study in [5] only records the beginning of all connections to reduce losses and save storage. The study in [11] improves this approach by recording only the beginning of all of the connections and the first part of packets.
Replay tools can be either stateless or stateful. Stateless replay tools send traffic according to the timestamps of packets. TCPreplay [6] can split traces to simulate the behavior between the server and the client through two interfaces. Tomahawk [7] is similar to TCPreplay, but it can retransmit packets when the packets are dropped. Stateful replay tools can keep the states of connections like TCP during replay. SocketReplay [11] maintains the TCP connection state by creating new socket connections. TCPopera [9] emulates the states for each TCP connection. Monkey [8] focuses on TCP replay, and it uses the socket interface to keep the connection state and simulate the delays in connections. The study in [10] focuses on the effectiveness of stateful replay, it can dynamically change the volume of generated traffic during replay. To improve the effectiveness of defect reproduction, we need to build a replay environment which is the most similar to the defect-triggering environment. Despite the above replay tools can be used for stateless and stateful devices, they are insufficient to reconstruct the replay environment for multi-port networking devices. Because these tools cannot split traffic during replay, they cannot reproduce some defects caused by the interaction of multi-port traffic such as the overloads of two different VLANs in a switch.