Network Simulation and Testing-Identifying Internet Dynamics

Network Simulation and Testing

Polly Huang EE NTU

http://cc.ee.ntu.edu.tw/~phuang [email protected]

Dynamics Papers

• Hongsuda Tangmunarunkit, Ramesh Govindan, and Scott Shenker. Int ernet path inflation due to policy routing. In Proceedings of the SPIE I TCom, pages 188-195, Denver, CO, USA, August 2001. SPIE

• Lixin Gao. On inferring automonous system relationships in the intern et. ACM/IEEE Transactions on Networking, 9(6):733-745, December 2001

• Vern Paxson. End-to-end internet packet dynamics. ACM/IEEE Trans actions on Networking, 7(3):277-292, June 1999

• Craig Labovitz, G. Robert Malan, Farnam Jahanian. Internet Ro

uting Instability. ACM/IEEE Transactions on Networking, 6(5):5 15-528, October 1998

Doing Your Own Analysis

• Having a problem

• Need to simulate or to test

• Define experiments

– Base scenarios

– Scaling factors

Base Scenarios

• The source models

– To generate traffic

• The topology models

– To generate the network

Internet Dynamics

• How traffic flow across the network

– Routing

– Shortest path?

• How failures occur

– Packets dropped

– Routes failed

– i.i.d?

Policy routing

Identifying Internet Dynamics

Routing Policy

Packet Dynamics

Routing Dynamics

To the best of our knowledge,

we could now generate:

AS-level topology

The Problem

• Does it matter what routing computation we

use?

• Equivalent of

Topology with Policy

• Internet Path Inflation Due to Policy Routing

• Hongsuda Tangmunarunkit, Ramesh Govindan, Scott Shen ker

• In Proceedings of the SPIE ITCom, pages 188-195, Denve r, CO, USA, August 2001. SPIE

Paper of Choice

• Methodological value

– A simple ‘re-examine’ type of study

– To strengthen technical value of prior work

• Technical value

– Actual paths are not the shortest due to routing policy. – The routing policy is business-driven and can be quite

hard to obtain.

– Shown in this paper, for simulation study concerning large-scale route path characteristics, a simple

shortest-shortest

Inter-AS Routing

Hierarchical Routing

Flat Routing

source destination

5:3

Hierarchical Routing is not optimal

Or

Prior Work

• Based on

– An actual router-level graph

– An actual AS-level graph at the same time

– Overlay the AS-level graph on the router-level graph

• Compute

– For each source-destination pair

– Shortest path using hierarchical routing – Shortest path using flat routing

Prior Conclusions

• 80% of the paths are inflated

• 20% of the paths are inflated > 50%

• There exists a better detour for 50% of the s

ource-destination pairs

– There exists an intermediate node i such that Le

ngth(s-i-d) < Length(s-d)

This Work

• To address 2 shortcomings

– There’s now a newer router-level graph

– There’s now a more sophisticated policy model

Newer vs. Older Graph

• Inflation difference not the same

– Difference is larger in the newer graph

– Due to the newer graph being larger

Shortest-AS vs. Policy-AS Routing

• Shortest-AS

• Policy-AS

– Not all ASs are the same

• Some are provider ASs • Some are customer ASs

Consider TANET CHT

Routing with Constraints

• Routes could be

– Going up

– Going down

– Going up and then down

Inferring the Constraints

• On Inferring Autonomous System Relationships in the Inte rnet

• Lixin Gao

• ACM/IEEE Transactions on Networking, 9(6):733-745, D ecember 2001

Not All ASs the Same

• 2 types of ASs

– Customer

– Provider

• 3 types of Relationships

– Customer-provider

– Provider-provider

Customer-Provider

• Formal definition

– A provider transits for its customer

– A customer does no transit for its provider

• Informal

– Provider: I’ll take any traffic

– Customer: I’ll take only the traffic to me (or my custom ers)

Peer-Peer

• Formal Definition

– A provider does not transit for another provider

• Informal

– I’ll take only the traffic to me (or my customers)

Sibling-Sibling

• Formal Definition

– A provider transits for another provider

• Informal

– I’ll take any traffic – You’ll take any traffic

Never “Going Down and then

Up”

• A provider-customer link can be followed by only

– (Or sibling-sibling link)

• A peer-peer link can be followed by only

Heuristics

• Compute out-degrees

• For each AS path in routing tables

– 1

AS with the max degree the root of hierarchy

– From the root, drawing providercustomer

relationship down 2 ends of the AS path

Determining Siblings

• After gone through all AS paths

• Any AS pair being both provider and

customer to each other are siblings

Determining Peers

• Do another pass on the AS paths in routing t

ables

• For each AS path

– Top AS who does not have sibling relationships

with the neighboring ASs

– Could have peering relationship with the higher

out-degree neighbor

– Given the Top AS and the higher out-degree nei

ghbor are comparable in out-degree

Back to Path Inflation

• Draw the customer-provider, peer-peer, and

sibling-sibling relationships on the overlay AS

graph

• Compute the best routes under the ‘never going

down and then up’ constraint

• Compare the inflation difference and ratio again

with these running at the inter-AS level

Shortest vs. Policy Routing

• Pretty much the same both in terms of

– Inflation difference – Inflation ratio

Therefore

• The observations from the prior work holds

– With a newer graph

Now forget path inflation

How far away is the shortest to the

policy inter-AS routing?

Shortest vs. Policy

• In AS hops

– 95% paths have the same length

– Policy routes always longer

• In router hops

– 84% paths have the same length

95% and 84% are pretty good numbers

Therefore shortest path at the

inter-AS level might be OK…

To Answer the Question

• Can we simply do shortest path

computation?

– A likely yes for AS-level graph

– A firm no for hierarchical graph

• Must separate inter-AS shortest and intra-AS shortest

Identifying Internet Dynamics

Routing Policy

Packet Dynamics

The Problem

• But how perfect is the Internet?

• The Internet

– A network of computers with stored information – Some valuable, some relevant

– You participate by putting information up or getting information down

At the philosophical level…

Humans are so bound to failures.

And the Internet is human-made.

But, Seriously…

Web Surfing Failures

• The ‘window’ waving forever?

• An error message saying network not

reachable

• An error message saying the server too busy

• An error message saying the server is down

• Anything else?

Network Specific Failures

• The ‘window’ waving forever?

• An error message saying network not

reachable

• An error message saying the server too busy

• An error message saying the server is down

• Anything else?

The Causes

• The ‘window’ waving forever

– Congestion in the network – Buffer overflow

– Packet drops

• An error message saying network not reachable

– Network outage

– Broken cables, Frozen routers – Route re-computation

Back to the Problem

• But how perfect is the Internet?

• Equivalent of

– Packets can be dropped

• How frequent • How much

– Routes may be unstable

• How frequent • For how long

Significance

• Knowing the characteristics of packet drops

and route instability helps

– Design for fault-tolerance

– Test for fault-tolerance

There are tons of formal/informal

study on the dynamics…

Packet Dynamics

• End-to-End Internet Packet Dynamics • Vern Paxson

• ACM/IEEE Transactions on Networking, 7(3):277-292, Ju ne 1999

Emphasis in Reverse Order

• Real subject of study

– Packet loss

– Packet delay

• Necessary assessment

– The unexpected

Measurement

• Instrumentation

– 35 sites, 9 countries

– Education, research, provider, company

• 2 runs

– N1: Dec 1994

– N2: Nov-Dec 1995

– 21 sites in common

Measurement Methodology

• Each site running NPD

– A daemon program

– Sender side sends 100KB TCP transfer

• Sender and receiver sides both

– tcpdump the packets

• Noteworthy

– Measurement occurred in Poisson arrival

• Unbiased to time of measurement

– N2 used big max window size

Packet Loss

• Overall loss rate:

– N2 higher, because of big max window?

• I.e. Pumping more data into the network therefore more loss?

• Big max window in N2 is not a factor

– Assumption: ack traffic in a half lower rate

• Won’t stress the network

Quiescent vs. Busy

• Definition

– Quiescent: connections without ack drops

– Busy: otherwise

• About 50% of the connections are quiescent

• For connections are busy

More Numbers

• Geographical effect

• Time of the day effect

Towards a Markov Chain Model

• For hours long

– No-loss connection now indicates further no-loss conne ction in the future

– Lossy connection now indicates further lossy connectio ns in the future

• For minutes long

– The rate remains similar p_n

No loss Loss

p_l 1-p_n

Another Classification

• Data

– Loaded data: packets experiencing queueing delay due t o own connection

– Unloaded data: packets not experiencing queueing dela y due to own connection

– Bottleneck bandwidth measurement is needed here to d etermine whether a packet is loaded or not

3 Major Observations

• Although loss rate very high (47%, 65%, 68%), all

connections complete in 10 minutes

• Loss of data and ack not correlated

• Cumulative distribution of per connection loss rate

– Not so exponential for ack

– Adaptive sampling contributing to the exponential obse rvation?

More on the Markov Chain Model

• The loss rate Pu

– The rate of loss

• The conditional loss rate Pc

– The rate of loss when the previous packet is lost

• Contrary to the earlier work

– Losses are busty

You might ask…p

_l

,p

_n

?

Values for the p

_l

’s

N1

N2

Loaded data

49%

50%

Unloaded data

20%

25%

Possible Invariant

• Conditional loss rate

• For the value remains relatively close over

the 1 year period

• More up-to-date data to verifying this?

• The loss burst size log normal?

Packet Delay

• Looking at one-way transit times (OTT)

• There’s model for OTT distribution

– Shifted gamma

– Parameters changes with regards to time and

path…

• Internet path are asymmetric

Timing Compression

• Ack compressions are small events

• So not really pose threads on

– Ack clocking

– Rate estimation based control

• Data compression very rare

Queueing Delay

• Variance of OTT over different time scales

– Divide the packets arrival into intervals of  – For all 2 neighboring intervals l, r

• ml the median of OTT in interval l

• mr the median of OTT in interval r

• Calculate (ml-mr)

Finding the Dominant Scale

• Looking for ’s whose queueing variance ar

e large

– Where control most needed

• For example, if those ’s re smaller than R

TT

– Then TCP doesn’t need to bother adapting to q

ueueing fluctuations

Oh Well

• Queueing delay variations occur

– Dominantly on 0.1-1 sec scales

Share of Bandwidth

Conclusions on Analysis

• Common assumptions violated

– In-order packet delivery

– FIFO queueing

– Independent loss

– Single congestion time scale

– Path asymmetry

Conclusions on Design

• Measurement methodology

– TCP-based measurement shown viable

– Sender-side only inferior

• TCP implementation

The Pathologies

Packet Re-Ordering

• Varying widely and too few samples

• Therefore, deriving only a

rule of thumb

– The Internet paths sometimes experience bad reordering

– Mainly due to route flapping

– Occasionally this funny case of router implementation

• Buffering packets while processing a route update

• Sending these packets interleaving with the post-update arrivals

Orthogonal to TCP SACK

• Receiver end modification

– 20 msec wait before sending duplicate acknowledgeme nt

– Waiting for re-ordered packets therefore lower false du plicate acknowledge

– Dup acks should be indication of losses

• Sender end motification

Packet Replication

• Very strange, can’t quite explain

– A pair of acks duped 9 times, arriving 32 msec apart – A data packet duped 23 times, arriving in burst

• False-configured bridge?

• Observation

– Most of these site specific

– But small number of dups spread between other sites – Senders dup packets too

Packet Corruption

• Checksum good?

• Problem

– The traces contain only the header data

– Pure ack OK, the header = the packet

– Data not OK, the header <> the packet

• Use an corruption inferring algorithm in tc

Corruption Rate

• 1 corruption out of 5000 data packets

• 1 corruption out of 300,000 pure acks

• Possible reasons of the difference

– Header compression

– Packet size

– Inferring tool discrepancy

Implication

• 16-bit checksum no longer sufficient

– A corrupted packet has a one 216th chance to have the s

ame checksum as the non-corrupted packet

– I.e., one out of the 216 corrupted packet can’t be detecte

d by the checksum

• Since 1 out of 5000 data packets is corrupted

– 1 out of 5000 * 216 (300 M) packets can’t be identified a

s corrupted by the TCP 16-bit checksum

Estimating Bottleneck Bandwidth

• The packet pair technique

– Send 2 packets back to back (or close enough)

• Inter-packet time, T2-T1, very small – When then go across the bottleneck

• Serving packet 1 while packet 2 will be queued • Packet 2 immediately follow packet 1

– Packets will be stretched

• Internet-packet time, T2-T1 , now the transmission time of

packet 1

This Won’t Work

• Bottleneck bandwidth higher than sending

rate

• Out-of-order delivery

• Clock resolution

• Changes in the bottleneck bandwidth

• Multi bottlenecks

PBM

• Instead of sending a pair

• Send a bunch

• More robust again the multi bottleneck

problem

Identifying Internet Dynamics

Routing Policy

Packet Dynamics

Route Instability

• Internet Routing Instability

• Craig Labovitz, G. Robert Malan, Farnam Jahanian

• ACM/IEEE Transactions on Networking, 6(5):515-528, O ctober 1998

BGP Specific

• BGP is an important part of the Internet

– Connecting the domains – Widespread

– Known in prior work that route failure could result in

• Packet loss

• Longer network delay

• Network outage (Time to globally converge to local change)

• A closer look at the BGP dynamics

– How much route updates are sent – How frequent are they sent

BGP

(In a Slide)

• The routing protocol running among the border

routers

– Path Vector – Think DV

– Exchange not just next hop, but entire path

• Dynamics

– In case of link/router recovery

• Exchange from the recovering point the route announcements

– In case of link/router down

Data Collection

• Monitoring exchange of route updates

– Over 9 month period

– 5 public exchange points in the core

• Exchange point

– Connecting points of ASs

– Public exchange: of the US government

Terminology

• AS

– You all know

– In the path of the path vector exchanged by BGP

• AS-PATH

• Prefix

– Basically network address

– The source/destination of the route entries in BGP

Classification of Problems

• Forward instability

– Legitimate topological changes affecting paths

• Routing policy fluctuation

– Changes in routing policy but not affecting

forwarding paths

• Pathological updates

– Redundant information not affecting routing

nor forwarding

Forwarding Instability

• WADiff

– A route is explicitly withdrawn – Replaced with an alternative route – As it becomes unreachable

– The alternative route is different in AS-PATH or next-hop

• AADiff

– A route is implicitly withdrawn – Replaced with an alternative route

In the Middle

• WADup

– A route is explicitly withdrawn – Then re-announced as reachable – Could be

• Pathological

• Forwarding instability: transient topological change

• AADup

– A route is implicitly withdrawn

– Replaced with a duplicate of the original route

• Same AS-PATH and next-hop

– Could be

• Pathological

Pathological

• WWDup

– Repeated withdraws for a prefix no longer reac

hable

Observations – The Majority

• Pathological updates (redundant)

– Minimum effect on

• Router processing load

– Some not agree

– Adding significant amount of traffic

Observation - Instability

• Forwarding instability

• Policy fluctuation

– AADup quite high

– But most probably pathological

Observation – Distribution

• No spacial correlation

– Correlates to router implementation instead

• Temporal

– Time the the date effect, date of the week effect – Therefore correlates to network congestion

• Periodicity

– 30, 60 second period

– For self-sync, mis-configuration, BGP is soft-state base d, etc

Basically, not saying much…

But for the background

And ease of reading

What Should You Do?

• Routing policy

– Intra-AS: shortest path

– Inter-AS: shortest path (95%, 84% OK) – Better model in progress…

• Packet losses

– 2-state markov chain model

• pl: some info

• pn: no info…