STP Flooding

[gregarican]

Every couple of weeks our LAN grinds to a near-halt. Trying to ping hosts on the local segments time out every other attempt.

Runnng Wireshark the last time this happened I see that there are tons of Spanning Tree for Bridges CTRL MAC Pause - Quanta 0 and 65535 packets throughout the packet captures.

My LAN doesn't have any managed switches. Only two unmanaged switches. The MAC address on these packets is just the generic STP one (01:80:C2:00:00:01). So I am having trouble tracking down the culprit. Over time these network congestion issues disappear. Like maybe 30 minutes or so the problem just vanishes.

Any tips for trying to get to the root cause? If I had managed switches I could check their event logs and ensure that STP was disabled. But this isn't an option.

Maybe just one bad NIC on a PC?

 

[gregarican]

Just some more supporting details. Both the source and destination MAC address is the generic 01:80:C2:00:00:01. That's what seemingly makes it tough to narrow down. If it was a rogue switch then I would assume the MAC address of the switch would appear somewhere in the captures. No such luck :-(

 

[gregarican]

One thing I'm doing is looking at some server NIC settings. A few of them have Flow Control set to Generate and Respond. After hours I am going to change the setting to Disable.

Since the switches are unmanaged I'm thinking that's perhaps why there's no "real" MAC address showing up in the CTRL MAC Pause packets being slammed out there.

Our network gateway is a dual-homed ISA 2004 box, and it has its NIC Flow Control set to Generate and Respond to these packets. Perhaps it's getting some static and propagating things?

Right now it's all a guessing game since the problem is very sporadic and there's no MAC showing up other than the generic STP one...

 

[MrNick0483]

You may want to start looking for a possible netowrk loop.

 

[gregarican]

I checked the wiring and the layout of things. No loops present. Checking things I did see a potentially vulnerable Linksys WRT54G wireless router that is attached to our Interent-facing unmanaged switch. If outside parties could tap into this then there's a chance they could've initiated random Ethernet pause flooding.

Since looking at this possible cause I locked down the Linksys further and have disabled Flow Control on most all of our servers. We'll see how things go.

 

[gregarican]

I happened again. For three weeks things were quiet. But again today the network was crawling. Packet captures showed these CTRL MAC Pause frames hitting the wire a couple of times per second. The Quanta 65535 and Quanta 0 packets back to back.

One of the segments showing this I tried swapping the hub out. Still no dice. After about an hour and a half the problem went away. I power cycled the other hubs and switches on the LAN during that time.

Wish I could trace the true sender of these packets :-/

 

[MrNick0483]

The next time it happens go to one of your main switches and sytematiclly start disconneting patches from it while running an active ping and wire shark. When everything goes back to ok you know you have found the problem and were to start fixing.

 

[gregarican]

That makes sense. Our network is made up of a basement floor with two 24-port hubs and an 8-port switch, and first floor with a 24-port hub and a 24-port switch. When these slowdowns happen it seems to span different network segments. The basement floor's PC's are slow as well as the first floor's PC's.

That's one fundamental question I had. Would these CTRL MAC Pause packets be propagated throughout all of the equipment? I guess if these are all hubs and unmanaged switches the answer would be yes.

So in turn I figure I have a lot of unplugging to do :-(

 

[MrNick0483]

Unless it is a "broadcast" or "unicast" traffic type it should not be propogating across all ports of your switches, only your hubs. Going back to basics, a hub broadcasts traffic to all ports to where a switch only forwards the data to the correct MAC destination on a particular port, thats why it is leading me to believe you have a bad NIC, network loop or a brodcast storm going on.

 

[gregarican]

True. I can't recall if CTRL MAC Pauses are broadcast type packets. Makes sense that switches wouldn't propagate them. Thinking back, the network slowdowns didn't affect hosts attached to the basement switch ports. Only hosts attached to the basement and first floor hubs.

If it is a bad NIC it will surely take some time and effort going through our attached to the various hubs. But it needs to be done. I use an app called Ntop that will monitor network utilization and any overly chatty hosts will throw a red flag I suppose...