ALL LIGHTS ON SWITCH BLINK NON-STOP

[hydruid]

Hello,

I have a flat vlan setup with 10, 3560g switches.

A week ago out of no where all the lights started blinking non stop as fast as they will go.

There are no physical loops. All switches are running the default config. I've shut each one down one at a time to see if that fixed the problem

Correctly asked questions will receive a better answer.

 

[Flyers01]

can you ping the switch
Is also connected to another switches on the network
could be a loop

 

[vipergg]

All it takes is a user to plug in a home switch on the net and then loop 2 ports together then bad things happen and it is a bear to find some of this stuff .

 

[VinceWhirlwind]

Get on your "core" and check your port stats: see which interface is seeig bucketloads of traffic.
Get on the switch that is uplinked to that interface and check its port stats to see where the crap is coming from. Disable the interface that it's coming from.

 

[hydruid]

the deal is that it's every switch in the network.

I'm convinced it's a hardware loop but I haven't been able to find it.

I installed zabbix tonight, to monitor the amount of traffic in and out of each port on each switch.

some of the ports are jumping up to 1000kbps of traffic in the middle of the night when no one is there, it will be crazy to see what it is during the day.

I'll go with Vince's advise and assume that vipergg is correct about someone hiding away a home switch

Correctly asked questions will receive a better answer.

 

[VinceWhirlwind]

If it's the middle of the night, you should be able to get away with disabling ports?

Disable every uplink off the cores and then enable them one by one until the #*it hits the fan. That should tell you where it's coming from.

Oh, and when you find the device, DON'T physically remove it from your network - get onto it somehow (preferably remotely) and assassinate it - if it's a PC, delete entire profiles and then *.* in c:\windows\system32\, if it's a switch administratively disable every port on it (except your uplink!), see if you can disable password recovery on it, then change its password.

The %*&#er who's just wasted days of your time will waste days of *his* time trying to figure out why his PC/switch doesn't work....it's only fair...

 

[hydruid]

I will do some more testing thursday but the problem is that stuff doesn't hit the fan right off the bat, it takes time for it to build up and cause the blinking lights

Correctly asked questions will receive a better answer.

 

[burtsbees]

Is this your new topology? When exactly in the process (if so) did this happen?

/

tim@tim-laptop ~ $ sudo apt-get install windows
Reading package lists... Done
Building dependency tree
Reading state information... Done
E: Couldn't find package windows...Thank Goodness!

 

[baddos]

The %*&#er who's just wasted days of your time will waste days of *his* time trying to figure out why his PC/switch doesn't work....it's only fair...

Wow thats pretty intense.

Back to the OP though, are you sure it's a loop? Cisco switches have spanning tree enabled by default. Check the configuration to make sure someone hasn't disabled it and definately make sure all the uplink ports are NOT set to portfast.

 

[hydruid]

I've set the switches back to default, so they're running a generic default config, which should be fine in a small flat vlan

I checked every port and every switch, took me a couple days but I did it, no hardware loops.

so portfast won't be anbled if I've set the switches back to default

The problem is that if I reboot everything at once, it doesn't happen all of a sudden, it takes an hour or two to start.

When I run wireshark I'm getting 15000 entries in 2 seconds. crazy stuff

Correctly asked questions will receive a better answer.

 

[unclerico]

do you have any multicast applications running??

I hate all Uppercase... I don't want my groups to seem angry at me all the time! =)
- ColdFlame (vbscript forum)

 

[robertjo24]

With a large flat network, there will be alot of broadcast traffic.

I have also seen problems with unicast flooding in these situations. I fought a problem with this last month. The Ciso document "Unicast Flooding in Switched Campus Networks
Document ID: 23563" was a little help.

 

[stubnski]

Has there been any new hardware/software added to your network?

When I first started working at my current job we allowed vendors to plug their laptops into our switches. One laptop almost brought down our network but I was not able to find out for certain what program was on the laptop to cause that issue although I do have some ideas.


Stubnski

 

[VinceWhirlwind]

Surely wireshark is giving you the source/destination of the bulk traffic?

 

[hydruid]

Updates:

I had a Ciso tech come out from my company and he looked around and gave me the following instructions:
1. block igmp at the router/firewall
2. enable bpdu port guard, and portfast for all switch ports besides uplinks
3. enable trunking on all switch uplinks

I have blocked igmp and do not see it anymore in my wireshark logs

I made the above changes on 19 switches yesterday, and have 10 more to go.

I will ready the document on unicast from cisco.

We have 6 servers, broken into two groups of three, that a network load balanced. I see a lot of traffic coming from both of the network load balanced ip's, 172.16.0.8 and 172.16.0.14, but not from the actual server ip's.

I'll keep you posted and any more ideas would be great.

Correctly asked questions will receive a better answer.

 

[hydruid]

I finally resolved the problem.

My windows cluster (network load balanced) that was sending out multicast igmp packets.

I changed the mode to unicast, and switched the value for MaskSourceMAC from 1 to 0 in the registry on the windows servers that were load balanced.

Here is the link where I found the information

http://technet.microsoft.com/en-us/library/bb742455.aspx

Correctly asked questions will receive a better answer.