Broadcasting Unicast traffic 1

[jkaftan]

I was sniffing on one of my networks and noticed a lot of traffic that I should not have been seeing. I was just sniffing in promiscuous mode with no port mirroring setup and I could see unicast traffic destined to another server. In a switched environment I do not expect to see that.

The server is on a different switch but in the same VLAN. One of my peers is telling me this is normal for a device that communicating with another segment. This is contrary to my understanding switching. Any thoughts on why the switch would be broadcasting this unicast traffic? Is it just because it is overwhelmed? There is a ton of traffic destined to this server

 

[burtsbees]

A switch will separate collision domains, but not broadcast domains. All devices in the same network, even traversing several switches, will see broadcast traffic. If they were separated by a router or VLANs, then they would become separate broadcast domains. Your colleague is correct.

Burt

 

[jkaftan]

Burt. Thanks for your reply and I agree that broadcast traffic would be seen by all on the same VLAN. However this is not broadcast traffic. This is typical unicast traffic i.e. TCP\IP communication between tow servers neither of which is the box I am sniffing from.

I looked in the CAM and I see no entry for the destination servers MAC. For other servers I see their MAC in the CAM and can see which port the traffic is being forwarded to. Usually it is a trunk port to another switch.

Even though this server is constantly receiving traffic there is no entry in the CAM. Thus the switch has to broadcast the traffic in order for it to get to the destination. The real question as I see it is why isn't the switch updating its CAM with this particular servers MAC?

 

[brianinms]

Your statements are contradicting, first you claim its unicast and then because the MAC isnt in the CAM you claim the switch is broadcasting to get to the destination. Are you sure its not a directed broadcast instead of unicast? What is this servers role?

 

[ADB100]

What you are seeing is 'Unicast Flooding'. This is very common in Campus networks where you have VLANs spanning multiple switches. You can tune the CAM & ARP timers to redunce it but you will never be able to elimiate it when you have VLANs spanned between access switches.

Have a read here to understand why and where you will see it.

http://www.cisco.com/en/US/customer...s700/products_tech_note09186a00801d0808.shtml

I have seen entire networks collapse because of this.... The best was a server backup over the network where only a couple of the switches had CAM entries for the destination server MAC address.
You can design this 'feature' out of your network but if you must have VLANs spanning your access switches then all you can do is tune the CAM & ARP timers to reduce its visibility.

HTH

Andy

 

[brianinms]

Good catch Andy, the main cause is asymmetric routing which is common in campus environments.

 

[ADB100]

Yes, the cause is usually Asymmetric routing, however if you follow the design guides and don't have spanning VLANs then even the traffic flooded from the alternate upstream distribution switch is only ever flooded down a single link to the access switch (L3 ARP entry but no L2 CAM). The access switch has the host directly connected so will have a CAM entry for the host. If another L2 switch was in the way and it didn't have a CAM entry, then traffic again gets flooded.

Andy

 

[jkaftan]

brianinms:

Sorry about the confusion there. I meant flooding when I said broadcasting.

We have a single 6509 at the core that is doing all of our routing for the LAN. I am hard pressed to see how it could be asymetric routing since we only have this one router. I am seeing the flooding on the 6509 and the server is on an access layer switch. I looked in the L2 table on the 6509 and the MAC of the server is not in there.

I wonder if the server has two nics. If the server was recieving traffic on one nic and sending it out on another then the 6509 would not see any retrun traffic from the MAC that it was sent on. That would cause the same issue as asymetric routing.

What do you guys think?

 

[ADB100]

It could be NIC teaming setup wrong, I have seen this before. You are right in what you have started from a troubleshooting point of view. Identify the L3 address, then the ARP entry on the router and then try and identify where it is at Layer-2. If the L3 switch has an ARP entry but no CAM entry then it will flood the packet to all ports in the VLAN.

HTH

Andy

 

[jkaftan]

That is what I did. There is an ARP entry but no L2. The server admin does not have Nic teaming in place. He just has two NICs with two seperate IPs. At least when I do a NS lookup with the server's name it returns two addresses. I would think with teaming it would just get one.

 

[ADB100]

How is the Layer-2 made up? Is the server connected directly to the 6500 or is it via a layer-2 switch? If it is on a layer-2 switch how is this connected to the router/layer-3 switch? You need to understand how the layer-2 and layer-3 elements work in the network, it is possible this is normal behaviour for the network you have.

Andy

 

[jkaftan]

The server is an anti-spam and AV server. All incoming email is sent to this server. The communication I am referring to is all email traffic.

The SpamAV is two switches from the core. See attachment.

I looked at the destination address of the flooded traffic I am seeing (10.3.0.211) and figured out what port on the access switch that NIC is attached to. I looked at the port (FE0/2) and the 5 min input rate is zero. I watched the input packets counter for awhile and see no change.

The input counter on the other port (FE0/1) is going up. So it looks like the SpamAV server is not sending traffic out to the FE0/2 interface but only to the FE0/1 interface. This would cause my problem.

The strange thing is that the server is ultimately talking to a device on the DMZ. The firewall would never allow the return traffic if it is comming from another IP. So the server must be sourcing the return traffic from the .211 address but sending it out the 210 interface. I have seen his servers do this before. It is really weird.

I did a sniff on the firewall port that faces the SpamAV server and I can see that the email server on the DMZ is communicating with this server on both .211 and 210 and getting responses on both 211 and 210. So I see no other explination then the server is accepting traffic on both NICs but only sending on the .210 NIC. Since the other nic .211 is not sending any traffic the switches cannot learn and update their L2 tables.


PS I was going to attach a drawing but it looks like I can only give a link to a web folder so I am going to pass. I hope this post makes sense without it.

 

[brianinms]

I would verify with your server team that each NIC is assigned an IP address and that the one NIC doesn't have 2 IPs assigned to it. The good news is that it seems you have isolated the issue.