Network Loops

[VOIPaintEASY]

Over the past 4-5 years since putting in dual core 8600’s and links from each 8600 to closet stacks, etc. We have been bitten by network loops. (These loops swamp the VLAN with BPDU’s end up knocking down the 8600 gig uplinks via Excessive broadcast CP limits, etc.) Loops in the form of upstream switches in conf. rooms that get looped. (BPBU filtering will block this on the 470 or 4500 switches) We had a scenario yesterday when ports on the same VLAN from different closet stacks got bridged in a conf. room and took that whole VLAN down. Spanning tree did not help across disparate switch stacks and BPDU filtering did not either. We have a campus of about 700 folks. We are by far not the biggest campus so I know other must face this on a larger scale than I. Events such as this take a whole floor down and have happened 2-3 times in the past 5 years. Any tips or advice.

 

[andy88]

i assume you are running smlt to edge.

Aswell as cp-limit you can use loop-detect (# conf ethernet ?/? loop-detect enable). This detects the same source mac being learn't on different ports and will shut one of them down. (Make sure you never enable it on IST ports)

If you are using 4.1 or higher the is a feature called slpp which also detects loops.

You could also enable stp on all ports on edge switches apart from the uplinks. This will prevent the cross linking issue.

 

[VOIPaintEASY]

I did have Spannigg tree on the edge switches but the loop came via 2 ports on same VLAN from diff. switch stacks. Are you talking Loop detect being able to block it at the gig SMLT uplink ports on the 8600's? Those ports have loop detect. Otherwise the Baystacks in the closets dont have Loop Detect options under VLAN settings.

 

[andy88]

Loop-detect is a feature only on the 8600, and should be enabled on the smlt uplinks on 8600. Never IST ports!!!

In theory then, if 2 different stacks are connected the 8600 will see the same source mac from 2 different ports(stacks) and shut one of them down.

 

[DaddyOfThree]

I would highly recommend that you look at SLPP (Simple Loop Protection Protocol) assuming you have the right software level on your ERS 8600 switches. I know of several very large organizations using it besides me and they totally love it.

Question though, how is that that you have a conference room that is cabled to two different closets/stacks? The usual problem people run into is a technician or end-user that puts a physical loop into the switch/stack in the closet or a user and mistakenly jumpers two data jacks together that feed back to the same switch/stack.

Here's a teaser;

http://blog.michaelfmcnamara.com/2007/12/simple-loop-prevention-protocol-slpp/

I believe there is a TCG (Technical Configuration Guide) available from Nortels' website that discusses the feature in detail.

You should also use rate limiting on your closet/edge switches/stacks and core uplinks to try and take the sting out of any loops. It might allow you time to locate the problem while operating at a decreased capacity.

Cheers!

 

[VOIPaintEASY]

We have a large enough floor with 250-300 folks on it so we have 3 different stacks in the closet for this floor. In trying to discourage hubs in conf. rooms my guys try to heat up multiple ports assuming the BPDU filtering would protect us. After 5-6 years and needing more ports at diff. times they have heated them up with different ports from diff. stacks. Guess not in this case. It get twice as messy in that we are 100% VOIP so we have way more hot ehternet ports over and above just data ports at many operations. When you talk about rate limiting are you meaning only the GIG uplink ports or the individual access ports? Thanks for the reply and you input.

 

[DaddyOfThree]

You might want to try and consolidate those individual stacks to a single stack (8 switches * 32 port = 384 ports per stack) although I know that's easier said than done.

You can rate limit the multicast/broadcast traffic on the edge/closet switches and on the uplinks into the core.

On the edge switches (5500s, 4500s, 470s) the following command will limit all multicast and broadcast traffic to 10% of the link utilization (you may want to experiment and try a value like 3%);

interface fastEthernet ALL
rate-limit both 10
exit

On the ERS 8600 it depends on the card your using to connect the closets but the commands to explore would be "rate-limit" and "broadcast-bandwidth-limit/multicast-bandwidth-limit".

Good Luck!

 

[VOIPaintEASY]

Bingo, I found that rate limit was set to 1% on switches #1 & 2 but disabled on 3-6. See config below. I will test this by replicating the loop tonight on switch 2 and then 3. 1/2 of the loop was plugged into switch 3 when it happened on this stack. I will report the results Friday. THanks for networking with me.


exit
interface FastEthernet ALL
rate-limit port 1/1-46 both 1
rate-limit port 1/47 both 0
rate-limit port 1/48,2/1-24 both 1
rate-limit port 2/25-26,3/1-24,4/1-24,5/1-24,6/1-26 both 0
exit

 

[edslopes]

Norel recomends boht CP Limit and EXT. CP Limit to protect your CORE against broadcast/multicast storms and DOS attacks.
CP Limit default values DO NOT protect your Core/network, try to find the Network Design Recomendations from Nortel and change it. You should enable rate limit on edge switchs/stacks too (Try 2% or 3%). Always remember, rate limit drops packets, cp limit lock down ports.