Performance Evaluation
5.4 Effects of Collaborative Routing Protocols on WMNs
5.4.6 OSPF with the Expected Transmission Count Metric
The Expected Transmission Count metric (called ETX) is designed for RoofNet [29]
at MIT to enhance the performance of the ad-hoc routing protocol. The ETX metric is used for a routing protocol to choose a routing path with a higher end-to-end throughput.
In the RoofNet, DSR is modified to work with the ETX metric and it is shown that using the ETX metric can improve the end-to-end throughput between a pair of nodes. To see how ETX can improve the OSPF routing protocol in WMNs, we implemented a version of OSPF with ETX support.
To support the ETX metric, each mesh AP records the number of hello packets it receives during a period and uses it to calculate the delivery ratio of each neighbor. The mesh AP stores these delivery ratios in its LSA packets to inform other mesh APs of the delivery ratios of its neighbors. These delivery ratios are used as link weights when a mesh AP computes the shortest paths to all other mesh APs. By this method, a smallest ETX path tree can be constructed, which can be used to find a high-throughput routing path between a pair of mesh clients.
5.4.7 Performance Evaluation
In this section, we evaluate the performance of WMNs using simulations. The simula-tions are performed on the NCTUns network simulator. Each case is simulated 20 times with different random mesh client locations. The average of these simulation results is presented in this section. The total simulation time of each case is 200 seconds. We only recorded the TCP results during the last 100 seconds to avoid the influence of the TCP startup behavior on the logged end-to-end throughputs. As Fig. 5.14 shows, 25 mesh APs are deployed in a 5x5 grid network. The distance between two neighboring mesh APs is
GatewayMesh Mesh Client
Mesh AP
Figure 5.14: The simulation topology 200 meters.
Each mesh AP has two IEEE 802.11(b) interfaces. One operates in the ad-hoc mode for forwarding packets among mesh APs and the other operates in the infrastructure mode for serving mesh clients. The transmission range and the interference range of these wireless interfaces are set to be 250/550 meters, respectively, which are used in the ns-2 network simulator [40] and commonly used in the literature. The ad-hoc mode and infrastructure mode interfaces use different channels to avoid interferences. The three routing protocols (i.e., OSPF, STP, and AODV) are run on mesh APs for performance evaluation. The mesh AP at the center of the field connects to the mesh Internet gateway via a link with the 100 Mbps bandwidth. There are 25 mesh clients scattered at random locations in the field. Each mesh client has an infrastructure mode IEEE 802.11(b) interface and uses it to send/receive packets to/from its associated mesh AP. For readers’ convenience, in Fig. 5.14 a mesh client is drawn at a location close to a mesh AP. However, in simulations since mesh clients are randomly placed in the field, the client-AP association relationship is not always one-to-one as shown in the figure. Multiple mesh clients can be associated with a mesh AP and a mesh AP does not have any mesh client associated with it.
Table 5.7: The total download throughput in the one-to-multi downlink traffic case
Protocol OSPF STP AODV
Total Download Throughput (KB/s) 253.8 255.7 245.6 Standard Deviation 15.3 14.2 16.4
Table 5.8: The number of established and stable connections in the one-to-multi downlink traffic case
Protocol OSPF STP AODV
Number of Established Connections 24.8 24.7 24.88 Standard Deviation 2.5 2.4 2.6 Number of Stable Connections 24.65 24.4 17.06
Standard Deviation 2.4 2.3 2
One-to-Multi Downlink Traffic Configuration
TCP is widely used in current Internet applications (such as, FTP, HTTP, email, etc.) to reliably transmit data across heterogeneous networks. Internet users usually download data from the Internet through an Internet gateway. Thus, we study a downlink TCP traffic case here. There is a TCP receiver (rtcp) running on each mesh client. Twenty five TCP senders (stcp) run on the mesh Internet gateway and each one greedily sends traffic to the TCP receiver running on each mesh client. That is, the 25 greedy TCP traffic flows compete for the bandwidth of the WMN. All mesh clients are fixed in this simulation case.
Tab. 5.7 shows the total download throughput of the WMN using the three evaluated protocols. one can see that the three routing protocols provide approximately the same total download throughput; however, as shown in Tab. 5.8, AODV results in the least number of stable connections in the WMN. (In this section, a TCP connection is defined as an unstable connection if the receiver program of this TCP connection does not receive data for Tsim2 seconds, Tsim is the total simulation time.)
An unstable TCP connection is due to excessive triggering of the TCP congestion control on the sending side of this connection, which prevents the TCP sender from sending out data for a long period of time. Due to the design of TCP congestion control, a packet loss will trigger TCP congestion control and many packet losses may result in a long transmission timeout.
To explain why AODV achieve the least number of stable TCP connections in a WMN, we studied its protocol design and the effect of its parameters. We found that in a
Table 5.9: Number of established and stable connections with AODV
ACTIVE ROUTE TIMEOUT 3 10 30 50 100
Number of Established Connections 24.88 24.85 24.5 23.95 23.2 Standard Deviation 2.6 2.5 2.4 2.5 2.3 Number of Stable Connections 17.06 16.6 21.6 21.7 22
Standard Deviation 2 1.6 2.1 2 2
highly-utilized WMN packet collisions happen quite frequently and in this situation TCP connections may constantly timeout for a long period of time. According to the design of AODV, if there is no traffic flowing on an established AODV routing path for a period of time (which is specified by the ACTIVE ROUTE TIMEOUT parameter), the source node will abandon the current path and re-flood the RREQ across the network to set up a new routing path. For this reason, when the TCP connection that uses the AODV routing path times out for a long period of time, AODV will abandon the used routing path and try to find a new one for the TCP connection. Flooding RREQ, however, consumes much network bandwidth and results in more packet collisions. In this condition, it is more difficult to establish routing paths and relay data packets. As a result, TCP connections more frequently trigger their retransmission timeouts and eventually become unstable.
The default the ACTIVE ROUTE TIMEOUT value is 3 seconds in AODV. As shown in Tab. 5.9, increasing the ACTIVE ROUTE TIMEOUT value can mitigate this problem and results in more stable TCP connections. However, although increasing the value of this parameter can generate more stable connections, it also causes AODV to respond more slowly to node movements. As will be presented later, this will result in a lower number of stable connections when mesh clients move in a WMN.
Tab. 5.10 shows the relationship between the hop count of the established connections and their achieved throughput in the OSPF routing protocol. As the average hop count of a connection decreases, the achieved throughput of the connection increases. This phenomenon shows that in a WMN “short” TCP connections usually can achieve more bandwidth than “long” TCP connections. We also studied the 20 runs of the OSPF case and found that if in a run there are more connections with fewer hop counts, the total download throughput of the run will be higher, as Fig. 5.15 shows. This phenomenon shows that using only the total download throughput as the sole performance metric to evaluate which routing protocol performs best may be misleading. A high total download throughput can be easily achieved by letting several “short” TCP connection monopolize
Table 5.10: Relationship between the average hop count and achieved throughput of connections
Hop Count 1 2 3 4
Throughput (KB/s) 14.33 11.19 9.12 7.6 Standard Deviation 3.2 3.1 3.7 2.7
100 150 200 250 300 350
0 5 10 15 20
Total System Throughput (KB/s)
Number of One-hop and Two-hop Connections
Figure 5.15: The relationship between the number of “short” connections and the total download throughput
the bandwidth of a WMN. Another important metric is the number of stable connections that can simultaneously exist in a WMN to share link bandwidth.
Multi-to-Multi Peer Traffic Configuration
In recent years, peer-to-peer applications are becoming more and more popular. Such applications include VoIP and music/video download applications. Here we created a simulation case to study the WMN performance with peer-to-peer applications. In this simulation case, each mesh client runs a TCP receiver (rtcp) and a TCP sender (stcp) and randomly sets up a greedy TCP connection to another mesh client. In total, there are 25 greedy TCP traffic flows in the system competing for the system bandwidth of the WMN. All mesh clients are fixed in this case.
Tab. 5.11 shows the total download throughput of the WMNs using this traffic pattern.
As one sees, the achieved total download throughputs are higher than those reported in Tab. 5.7. Note that in the previous downlink TCP traffic case, all TCP traffic flows need to merge at the single gateway. For this reason, they are bottlenecked at a single point in the WMN. In this multi-to-multi peer traffic case, however, all TCP traffic flows need not merge at the single gateway. Instead, they can choose the best shortest routing paths
Table 5.11: The system total throughput in the multi-to-multi peer traffic case
Protocol OSPF STP AODV
Total Download Throughput (KB/s) 756.9 569.8 603.1 Standard Deviation 32.2 22.3 25.6
Table 5.12: The number of established and stable connections in the multi-to-multi peer traffic case
Protocol OSPF STP AODV
Number of Established Connections 21.68 20.25 16.94
Standard Deviation 2.3 2.2 1.5
Number of Stable Connections 11.79 5.8 8.56
Standard Deviation 2.1 0.6 1.6
in the WMN for relaying their data. The freedom of routing will improve the efficiency of wireless bandwidth usage in the WMN.
Tab. 5.11 shows that OSPF provides a higher download throughput than AODV and STP. The reason why OSPF outperforms STP is that in STP a routing path between two mesh clients may not be the shortest one because the routes determined by STP should follow the tree structure. As a result, the bandwidth of the WMN may not be efficiently utilized by STP. The reason why OSPF outperforms AODV has been explained before.
In AODV, the active route timeout of a routing path is constantly triggered, which causes the RREQ to be flooded to the network, wasting the wireless bandwidth of the WMN.
Tab. 5.12 shows the number of established and stable connections of the WMNs with the three evaluated routing protocols. As can be seen, OSPF results in more stable connections than AODV and STP. The reason for AODV has been explained above. Here we explain the reason for STP. It is clear that, because packets can only be relayed on the spanning tree, STP may waste the network bandwidth due to the use of non-shortest-path route between a pair of mesh clients. Our simulation results show that a packet on average needs to traverse 3.99 hops to reach its destination client in STP while this number can be reduced to only 3.45 hops in OSPF. Due to this reason, given the same level of client traffic load, the level of congestion in STP is more severe than in OSPF.
This means that more packets will be dropped in STP, which include the control packets of STP. As a result, the spanning tree constructed in STP may need to be constantly repaired or changed. However, this will cause TCP connections to timeout more often and make them unstable connections.
Table 5.13: The system total throughput under the mobility condition
Protocol OSPF STP AODV
Total Download Throughput (KB/s) 243.2 244.1 230.8 Standard Deviation 14.8 14.9 13.2
Mobility Condition
In this section, we study the performance of the WMNs using the three evaluated routing protocols when mesh clients move. The settings of the used simulation case are the same as those of the downlink TCP traffic case, except that in the used case all mesh clients move at the speed of 1 m/s based on the random-waypoint mobility model. This model was first used by Johnson and Maltz in the evaluation of Dynamic Source Routing (DSR) [41], and was later refined by the same research group [42]. The refined version has become the de facto standard in mobile computing research.
As shown in Tab. 5.13, the WMNs using the three evaluated routing protocols achieve almost the same download throughputs. However, as can be seen in Tab. 5.14, AODV results in the least number of stable connections among the three evaluated routing pro-tocols. This phenomenon is explained here. When a mesh client changes its associated mesh AP from the old AP to the new AP, its original routing path is no longer valid.
Using AODV, however, the mesh APs on the original routing path need to wait for a long time before detecting such an event. Since the waiting time is long and a mesh client con-stantly changes its associated mesh AP while it moves, the disruption to a mesh client’s connection is too long and too often, which makes many mesh clients’ TCP connections unstable. In OSPF, when the new mesh AP gets the IEEE 802.11(b) association control packet from the mesh client, it broadcasts a LSA to inform other mesh APs that the mesh client now is associated with itself. If the old and new mesh APs are within each other’s wireless transmission range, the LSA can reach the old mesh AP very quickly.
This enables the old mesh AP to promptly forward the mesh client’s packets to the new mesh AP and shortens the period of disruption to the mesh client. The details are shown in Fig. 5.16.
In STP, a similar mechanism is used to deal with node mobility. In STP, when the new mesh AP gets an association packet, it broadcasts a control packet up the spanning tree to inform upper-level mesh APs of this association change. When the control packet reaches an appropriate layer in the spanning tree, the packets destined to the mesh client
Table 5.14: The number of established and stable connections under the mobility condition
Protocol OSPF STP AODV
Number of Established Connections 24.9 21.5 23.8 Standard Deviation 2.7 2.1 2.1 Number of Stable Connections 24.1 20.4 3.1 Standard Deviation 2.2 1.9 0.4
Figure 5.16: The handling of node movement in OSPF
will be directed toward the right branch of the spanning tree, which ends the period of disruption to the mesh client. This process is depicted in Fig. 5.17.
Multi-Gateway WMN
The total download throughputs of WMNs with a different number of mesh gateway nodes (i.e. those mesh APs connecting with the Internet gateway). These mesh gateway nodes are deployed at the corners of the simulated grid topology. Fig. 5.18 shows an example topology of a WMN with two mesh gateway nodes. OSPF is used in the cases in this section. The traffic settings of these cases are the same as those in the downlink TCP
Mesh AP
Figure 5.17: The handling of node movement in STP
GatewayMesh Mesh Client
Mesh AP
Figure 5.18: The network topology of a two-gateway 5x5 grid WMN
Table 5.15: The system total throughput of a multi-gateway WMN with different number of gateways
Number of Gateway 1 2 3 4
Total Download Throughput (KB/s) 254.1 381.7 670.4 903.1 Standard Deviation 15.5 17.6 28.4 39.6
traffic case, except that now multiple mesh APs connect to the mesh Internet gateway rather than just one. Tab. 5.15 shows the simulation results of these cases. As one sees, using more gateways in a WMN can significantly improve the total download throughput when most traffic in the WMN is Internet traffic. These results suggest that enough gateway nodes should be deployed in a WMN to make its performance scalable with the number of mesh clients.
OSPF with ETX
Here we study the effect of ETX when it is combined with the OSPF routing protocol.
The traffic settings of the studied cases are the same as those of the downlink TCP traffic case. To let ETX show its capability in harsh wireless environments, the two-ray ground model with the Rayleigh fading [43] is used in this simulation case. The bit-error model for binary phase-shift keying (BPSK) modulation is adopted to calculate the bit error rate (BER = 1/(2 ∗ (1 + P ower)). For each received packet, its received power is calculated based on the distance between the source and destination nodes. The received power is
Table 5.16: The system total throughput under OSPF and OSPF with ETX in a harsh wireless environment
Protocol OSPF with ETX OSPF
Total Download Throughput (KB/s) 128.1 180.5
Standard Deviation 10.5 13.9
then added with a random fading with a variance of 10 dbm. Based on the resultant power, the Bit Error Rate (BER) for the received packet is calculated. The received packet is then dropped with a probability based on the calculated BER.
Tab. 5.16 shows that OSPF with ETX generates a lower total download throughput than OSPF in this harsh environment; however, Tab. 5.17 shows that OSPF with ETX allows more stable connections to coexist than OSPF in this harsh environment. The reason is that in this harsh environment the shortest paths selected by OSPF are usually with high BERs and thus fragile. In this condition, most TCP connections with high hop counts are broken (in the TCP timeout state) and contribute little to the total download throughput. As a result, the total download throughput is mostly contributed by those TCP connections with low (2 or 3) hop counts. That is, a few “short-distance” TCP connections (initiated by those mesh clients that are close to the gateway mesh APs) monopolize the network bandwidth and block out many “long-distance” TCP connec-tions (initiated by those mesh clients that are far away from the gateway mesh APs).
Because the RTTs and hop counts of these “short-distance” TCP connections are small, the TCP congestion control of these connections allow them to rapidly pump their data into the WMN (due to the well-known “self-clocking” property of TCP), even when they experience some packet losses.
In contrast, using the ETX metric in OSPF helps OSPF choose a low-BER and higher-throughput routing path for a connection. Thus, “long-distance” TCP connections be-come more robust and more of them can achieve a high throughput. However, the cost of this more even sharing of system bandwidth among “short-distance” and “long-distance”
TCP connections is the reduced total download throughput. This is because the “long-distance” TCP connections in OSPF with ETX cannot react to packet losses as rapidly as the “short-distance” TCP connections in OSPF.
Table 5.17: The number of established and stable connections under OSPF and OSPF with ETX in a harsh wireless environment
Protocol OSPF with ETX OSPF
Number of Established Connections 24.75 24.8
Standard Deviation 3.1 2.9
Number of Stable Connections 22.7 6.75
Standard Deviation 3.1 2.3
Table 5.18: The total download throughput of single-radio and dual-radio WMNs
Protocol OSPF (1c) OSPF (2c)
Total Download Throughput (KB/s) 120.8 254.3
Standard Deviation 9.6 20.3
Single-Radio WMN vs. Dual-Radio WMN
One advantage of dual-radio-dual-mode WMNs over single-radio WMNs is that client-AP traffic and client-AP-client-AP traffic can be transported over different frequency channels at the same time to increase the total network throughput. To see how the second radio improves the total network throughput, we conducted two tests. In the first test, the ad-hoc mode and infrastructure mode interfaces of a mesh AP are set to use different channels. In contrast, in the second test, these two interfaces are set to use the same frequency channel. Because IEEE 802.11(b) MAC employs a carrier-sense multiple access
One advantage of dual-radio-dual-mode WMNs over single-radio WMNs is that client-AP traffic and client-AP-client-AP traffic can be transported over different frequency channels at the same time to increase the total network throughput. To see how the second radio improves the total network throughput, we conducted two tests. In the first test, the ad-hoc mode and infrastructure mode interfaces of a mesh AP are set to use different channels. In contrast, in the second test, these two interfaces are set to use the same frequency channel. Because IEEE 802.11(b) MAC employs a carrier-sense multiple access