Controlling Multicast Traffic (8600, 450, 470, 4548, 5520)

[rjenk]

I need some insight on dealing with multicast traffic on a multi-VLAN network.

I have two 8600's (v 3.7.0.0) in the core providing connectivity to a variety of closets with one of the following setups:

- BPS2000/450 Stacks (2- BPS2000s, 1 or more 450s)
- 470 Stacks
- 4548 Stacks (slowly replacing the above)
- 5520 Stacks (slowly replacing the above)
- 8100
- 8600 (in the computer room) - 3.7.0.0

The 8600 in the computer room is connected to the two core 8600's using SMLT - 3 links. The majority of the other closets use single uplinks but a few (less than 10) use SMLT.

There are about 60 VLANs (each stack is a VLAN). I have two non-routed VLAN's that are used for a group of equipment that uses multicast (VLAN75 and VLAN76). They only communicate to devices on their specific VLAN and have no need to route any type of traffic. We have another VLAN used for our servers (VLAN100). I am seeing a lot of multicast traffic on this VLAN and do not know the source. This is becoming an issue.

We are not using Snooping or any type of multicast routing.

What alerted us to the multicast traffic was that earlier this week, the 8600 in the computer room went offline. We force a restart and it began working again. We monitored it and found that the box is shutting down one port every few hours due to multicast packet floods. Now when it happens, we can disable the port and re-enable it and everything is fine. That started our search for multicast and we have found flooding of packets to all ports on VLAN100. There are three other stacks in the computer room that have VLAN100 (a 470 stack of 3 switches, a 5520, and a 5530/5520 stack).

How do I get this under control? Do I just enable Snooping on VLAN100 on all the switches with that VLAN? Since this traffic is not routed do I need to do something like DVMRP? I am unsure of how to proceed to 1. stop the links from going down and 2. to control the multicast traffic and stop it from being flooded.

Sorry for the lengthy post.

 

[lobk]

To ease the problem just enable rate limit for multicast trafic on the port.

Then try and track down the problem. I'm almost willing to bet there is a loop, like one of the vlan got bridge over.

 

[anthonyanderberg]

I don't have a step by step process, but maybe if I talk in a general way it'll get you started until one of the other wizards in the group has a chance to respond.

One thing that comes to mind is that there were a lot of features introduced in the 3.7 code branch, and you may be running into a bug. If you're at 3.7.0.0 you should consider upgrading to a later build in that branch or to a 4.1.x version, as mentioned in a post last week a 4.1.x upgrade would have memory and configuration requirements. I seem to recall lots of IGMP/DVMRP work/fixes being done in that time frame as well so an upgrade might really be worth considering if you turn on those protocols.

Is it the same port shutting down each time? In the logs can you see what kind of shutdown is the port doing? If its a CP-Limit shutdown, that is the 8600 trying to protect itself from what it considers to be a flood... you might get such a thing if you had an uncontrolled loop, although a couple people have posted with legitimate traffic that amounts to enough to cause a cp-limit shutdown... but that's rare and if you have that much legitimate multicast traffic you'll want to understand its flow. By the way, if the traffic is ligit you can tune those cp-limit thresholds or even disable cp-limit - at the risk of the 8600 CPU getting bogged down by too much traffic.

So understanding the source/purpose of this multicast traffic is probably the best step 1, and I would work that angle before doing any upgrades or config changes. Have you been able to use Ethereal/WireShark to get a sample of it? If it isn't continuous you could try port-mirroring the affected port with a large capture buffer to catch the last traffic the port has seen before if goes down.

We in the networking field often end up being part-time application and server folks as well, and it can be both a blessing and a curse... best of luck!

 

[rjenk]

Thank you for the suggestions.

We have put sniffers on several of our suspect VLANs to get a better picture of what is going on, although I have not seen much broadcast based traffic so far. I have bumped the cp-limit a little to give us some working room.

It almost always begins with the same port (2/2), then if not re-enabled it progresses to (2/1) and finally (1/2).

It had been happening 3-4 times per day so I am leaning away from it being a loop.

We are scheduled to do a major core upgrade week after next (upgrading the 8600's in the core with new boxes - 8692SF with SMezz, 10G blades) and will be upgrading the 8610 in the computer room with 8691SF's (from the old core) and taking it to 4.1.x.x code. Eventually we will pull the computer room 8600 as it is being replaced with 5530/20 stacks with 10G uplinks. So, maybe the changes will help if it is an code issue.

 

[anthonyanderberg]

Be sure to check the 8691s memory, some of the older CPUs need a DRAM upgrade to run newer code images. I've been using the 8692 with 10G and other R-series modules since their release, except for some early 4.0.x bugs they've worked well.

Are 2/2, 2/1, 1/2 all in an MLT? That might explain why its progressing from port to port and always starts with the same port. Until you can catch it in the act it'll be hard to really know what's happening... but of course management never likes to hear that the network will be down for another 15 minutes while we take traces and brainstorm.

From my experience loops only take a short amount of time to go from normal traffic to cp-limit/STP shutdown or network meltdown, and if the loop isn't fixed re-enabling the ports will just re-create the problem. You didn't mention it but I assume there isn't any pattern to the times the issues occur (ie: every time Fred starts a backup the storm starts.)

Rate limiting is a good idea and may buy you some time, but it won't know which packets to toss - it might drop packets other applications need. Of course having the switches lock up isn't great for application performance either...

 

[DaddyOfThree]

You may want to make sure that you enable Spanning Tree on all your edge ports on the ES470, ERS4500s and ERS5500s. The Auto Midi-X feature can really burn you if someone mistakenly patches two ports together.

Without IGMP snooping the multicast packets will get flooded out to all ports in the VLAN regardless if they are participating in the multicast traffic or not. You may want to enable IGMP snooping on the edge switches. You may also want to enable DVRMP within your ERS8600s. With DVMRP enabled you'll be able to see who's sending the multicast traffic.

You'll also want to make sure that the cp-limit feature is disabled on any IST links.

There are some examples of setting up DVMRP here;

http://blog.michaelfmcnamara.com/2008/03/multicast-routing-protocol-part-1/

Good Luck with the upgrade!

 

[rjenk]

Is the default setting for cp-limit okay for links to the core? By default it is set to 10,000 broadcast and 15,000 multicast.

Also, what effect with DVRMP have on overall traffic? Would the two isolated VLANs (no IP routing) that contains mostly multicast traffic be routed to other VLANs - their multicast traffic that is?

Thanks for the assistance everyone.

 

[DaddyOfThree]

In my environment I actually use half the default values, 5,000 broadcast and 7,500 multicast. I've seen occasions where the network will go belly up before cp-limiting kicks in with the default values. If you're running high-bandwidth multicast applications you'll need to test for yourself and find a suitable value.

DVMRP would only function on those VLANs that you enable it on. But it would allow you to use the ERS8600 to identify the sources (IP addresses) of the multicast streams. With that said you can add DVMRP unless you already have an IP interface in that VLAN. From your comment "no IP routing" I can only assume that you don't have an IP interface in that VLAN.

You could also use a packet sniffer such as WireShark and just sniff the packets to find the source IP address. Just connect the PC to any port in that VLAN and the multicast should be flooded to all ports, no need for any port mirroring.

Good Luck!