It is obvious that real-time capture and replay testing is effective, but it is not suitable for multi-port devices because of the limited number of ports in the bypass switch and the replay tools. The OF switch is a good choice for traffic handling like bypassing the DUT and splitting during replay. The purpose of this work, namely OFCR, is to combine the real-time capture and replay testing with the OF switch to construct an effective testing for multi-port DUTs. This chapter highlights the terminology and assumptions, and then discusses the problem statement.
3.1 Terminology and Assumptions
In this work, the term traffic is defined as dynamic packet flow in network, and trace is the static file which records the packet flow. A trace that records the traffic causing failures is a defect trace. Not all defects can be reproduced because some defects are due to a non-reproducible scenario like race condition. For a defect trace with reproducible defects, we call it a defect-triggering trace. Here we classify reproducible defects into two types:
overload defect and protocol defect. Overload defect is caused by a busy condition in the DUT such as hardware overload and table overflow, while protocol defect is triggered by specific content in packets, such as too short or too long payloads and content anomaly.
The procedure in OFCR can be divided into three stages: live mode, live-to-replay failover and replay mode. Most of the time in OFCR is in the live mode. It records the defect traces when the DUT fails from a normal state. In order to save the storage, we set thresholds to limit the number of packets and maximum packet lengths when capturing traffic.
Live-to-replay failover is a transition between the live mode and replay mode. It changes from live mode to replay mode when the network breaks down due to the DUT failure. It modifies the flow table in the OF switch to keep the network alive and deploy a multi-port replay circuit. Similarly, it can change from the replay mode to live mode when the DUT recovers from the failure. In the replay mode, OFCR replays the defect trace to the multiple ports in the DUT. If the DUT fails again after replay, then the trace is a defect-triggering trace and OFCR will process hybrid defect reduction. Hybrid defect reduction contains overload defect reduction and protocol defect reduction. The former assumes the defect is an overload defect
to operate reduction and the latter assumes the defect is a protocol defect. After these two reductions finish, we can derive the minimum defect-triggering trace.
Table 3 is the descriptions of the notations used in this work. There are three types of ports in the OF switch. Di represents the port i to the DUT port Ui, Pi represents the port i to live network and R1, R2 denote the ports to the two-port replayer. T denotes the defect trace in traffic capture with total count of packets (connections) c and max packet length l. Inside T, ti,j
represents the last j-th packet (connection) to Ui. When the layer of the DUT is less than four, the unit is packet, otherwise the unit is connection. uci is the count of packets (connections) to Ui . The trace used for replaying the defect, Tr,is derived from T. Some packets in T are incomplete because their original length is larger than l. This will reduce the effectiveness of replay for incomplete packet drop by the network interface, so we process checksum recalculation for packets in T by packet modification tools [14, 15] to derive Tr. When OFCR operates the hybrid defect reduction, reduced trace To, Tp and Tmin will be generated. The notations in the reduction part are used during the hybrid defect reduction.
Table 3: Description of notations.
Categories Notations Description
DUT N The number of ports used to connect network in the DUT
Ui Port i used to connect network in the DUT To Reduced trace by overload defect reduction
Tp Reduced trace by protocol defect reduction Tmin Minimum trace after reduction
Reduction
Tin Input trace of packet/payload reduction Tout Output trace of packet/payload reduction cini Packet (connection) to port Ui in Tin
couti Packet (connection) to port Ui in Tout
lin lout Max packet length in Tin Max packet length in Tout headi Index of the first packet (connection) to Ui in Tr
taili Index of the last packet (connection) to Ui in Tr
cut p Cut unit in packet reduction Cut unit in payload reduction tc tp Threshold in packet reduction Threshold in payload reduction
3.2 Problem Description
We will face some problems for debugging when dealing with a trace captured during a period of time before the DUT failure. First, the packet count c and the max packet length l of trace T may be insufficient to trigger the defects. Second, defects may not be triggered because the two-port replayer cannot forward traffic to multiple ports Ui like the original defect-triggering scenario. Finally, even though the captured defect trace Tr is small, it is not easy to identify the defect-triggering packets in Tr.
The detailed problem description is given as follows. Given a DUT with N ports connecting to network by an OF switch and a trace T captured when the DUT fails. The objectives of our work are (1) finding out minimum c and l in T that can triggers defects, (2) replaying packet ri,j in Tr to multiple ports Ui on the DUT, and (3) deriving the minimum defect-identifying trace Tmin from Tr.