strange ping response patterns

[jimfixit]

Whilst trying to diagnose dropped packets to the other side of our firewall, I have noted some strange behavior that I don't know how to account for.

Topology:

Workstation A --> 3750 --> 6513 --> 3500 --> firewall
|_> laptop B

When I use Workstation A to ping to each device in the route I see the following:

WSA to 3750 <1MS steady
WSA to 6513 <1MS steady
WSA to 3500 usually 1-5MS frequently spikes to 26 or 30 MS
WSA to firewall < 1MS steady
WSA to laptop B <1 MS Steady

What is is about that 3500 that, when pinged directly, would show these spikes in response but, when pinging through that same 3500 to a unit connected to it, does not display those same frequent spikes?

If I get spikes in response going to the unit, should I not see that overhead added into the response for devices connected to that unit???

Very odd.

 

[chipk]

If you're using QoS, or any kind of traffic shaping for that matter, a ping's response time is not going to tell you much. To take a guess about why this is happening, though (and someone who really knows the internals of these switches could confirm or deny), but probably what is happening is that if you're pinging through the device, the packets are being fast switched in ASICS; if you're pinging the device itself, it has to be processed by the CPU, which would take longer.

(steps aside and waits for someone to raise the bull**** flag).

 

[vipergg]

You can;t use ping response to a network device for anything more than a rough baseline .Ping response is low man on totem pole as far as things the switch will do . that is not out of the ordinary . The way you test is to ping thru a device to another pc off that device not to the network device itself.

 

[chipk]

Viper, not that I'm doubting you, but could you point to some documentation for what you state above? I know that packet priority is specified in the IP header, but if there is no packet prioritization enabled on the device, does the switch still move certain packets to "low-man" status?

 

[Minue]

Hello
Which device is the switch default-gateway?
In any case think this behavior is normal.The switch will pass the packets to firewall and laptop B at layer 2.But when responding at layer 3 to ip packets it's slower bacause switches aren't built to handle layer very well.
Regards

 

[chipk]

...which is what I was saying about the ASICS vs. processor.

That answer makes the most sense to me, but I would like someone to confirm or deny what Viper said. Would be good to know.

 

[burtsbees]

What response do you get pinging the firewall?

Burt

 

[Minue]

Hi Burt
I think this post is dead!No response from the poster in 3 days.
Regards

 

[burtsbees]

Oh yeah...ha ha. Thanks.

Burt

 

[jneiberger]

vipergg is absolutely right. ICMP and SNMP are low priority for Cisco network devices. If the administrative processes are busy doing something else then you might see a delay in (or lack of) responses. This is completely normal.

 

[Minue]

Hello jneiberger
Is this info documented in some where?

Regards

 

[chipk]

I'm going to go out on a limb and say this claim is not true. Unless you implement a queuing strategy or QoS, your network devices aren't going to treat a ping packet any differently than they will treat an SMTP, Telnet, DNS or any other packet.

I believe the default queing strategy on interfaces is FIFO anyway, which means calls are taken in the order they are received...

 

[jneiberger]

chipk, you're are wrong about this. We're talking about ICMP to the administrative interface on the switch. We are not talking about frames passing through the device. It is quite common for network devices to rate-limit ICMP to the administrative interface. I seem to recall that Cisco IOS devices typically rate limit ICMP to 2 per second, but I don't know how easy it would be to find documentation stating that. I'll look for it and let you know if I find it.

 

[jneiberger]

Here are a few links I found that mention the ICMP rate-limiting behavior in Cisco IOS:

http://ioshints.blogspot.com/2007/10/mturoute-trace-mode-output.html

http://www.velocityreviews.com/forums/t33605-icmp-myth.html

It's hard to find good info on this because the necessary search terms are so common. But this is well known behavior, especially if you've done a lot with network management tools that poll these devices. It's always better to ping through the router/switch to a device on the other side. Then again, this might not be as true now as it was five or six years ago when processors were much slower and these devices were easily overwhelmed.

 

[chipk]

You obviously misunderstood my original post, because what you say in the second to last post is exactly what I said. A packet traversing a switch goes through ASICs. If you're pinging an interface, the CPU comes into play, which is not the fast part of a switch, hence the delay. My statement still makes more sense to me, but if you can actually prove that I'm wrong instead of just pointing to other forums and blogs, I'll be glad to admit that you're right.

 

[jneiberger]

chipk, in reading back, your first post is correct. The processor has to get involved in order to process packets sent directly to it. Since these devices typically haven't had very fast processors, it would be easy to launch a DoS attack on them if they didn't rate-limit traffic. Cisco IOS devices typically rate-limit traffic to the administrative interface as a self-protection mechanism.

I'm sorry that those were the only two links I could find quickly. I don't have the time this afternoon to spend on this, but perhaps I'll look some more later on this evening.

 

[burtsbees]

I would be willing to bet that that if he tilted the 6513 toward the 3500, the electrons would go faster due to gravity, and his ping responses would be normal. Right now, they're likely going uphill.

Burt

 

[brianinms]

Aight I have to ask why you have an old 3500 between your firewall and your 6513. Or is it actually a 3550?

 

[burtsbees]

Perhaps that is the problem...they are getting dropped packets on the other side of it...

Burt

 

[jimfixit]

Thank you all for the insightful information. The post wasn't actually 'dead' but, it was one of those things that was hot until something else came a long. We had pre-scheduled time frame to massively rework all the MPLS network configs on 6 offices and our main HQ including two service providers for a total of 14 router reconfigs, 7 core switch reconfigs and a ton of fall out.

The problem with the dropped packets turned out to be due to a program on the internal network trying it's best to hammer away on some device out in internet land (to obtain updates to itself) and it just wouldn't say "die" when the device in internet land wouldn't respond. The software just kept hitting harder and harder...usurping all our firewall's ability to keep up. So other packets dropped. I say that's a virus behaivor not a legitimate update program, at the very least not polite.

I got side tracked by the speed differential in the ping diagnostic process but, all the data you folks have provided gives me wonderful insight for the next time. I won't take what I see as the gospel truth, nor will I waste time suspecting equipment when I need to be sniffing packets. Had I done that first I'd have seen this unit right away.

On another note, why the 3500? You got me. I can make recommendations all day long but I can't make the client do anything. As I understand, It went into play as a means to allow the security team which is different from the networking group, to have a switch they could control for session monitoring of internet traffic, thus allowing them to run various IDS and internet patrolling programs without having to give them access to the 6513. I would have picked a beefier switch (or set up a handful of ports and said, USE THESE).

Thank you again. All very good info.