Seeing all network traffic on all ports-why? 1

[hinesjrh]

A couple of weeks ago I began having complaints of phone calls (VOIP) breaking up and / or voice cutting out. My phones are on the same VLAN as the majority of my clients for flexibility. Yesterday I began to dig into this issue and see what is going on with my network. When we ran OmniPeak we began to realize that no matter where we are plugged into our network we are seeing all traffic - you would think we were running hubs instead of switches. Why?

My core switch is a Cisco 4006 running CAT OS and my edge switches are Cisco 2950's. I began to look at my 4006 this morning and I see when looking at the CAM, I am getting the following warning message. What is this telling me (port 2/44 is a span port for purposes of Websense)?

Console> (enable) show cam count dynamic
2006 Apr 26 10:27:50 %SYS-4-P2_WARN: 1/Filtering MAC address 00-00-00-00-00-00 o
n port 2/44 from host table
Total Matching CAM Entries = 761

What is a reasonable amout of CAM entries to expect? Is there a suggested aging besides the default of 300 seconds?

 

[ADB100]

The most probably answer is Unicast Flooding:

http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note09186a00801d0808.shtml

This is one of the biggest issue with flat networks (or networks with VLAN's spanned over your whole campus). You can tweak the ARP and CAM timers so it is less of an issue but you will never get rid of it if unless you have a proper hierarchical Layer-3 network. If you span your Layer-2 networks over multiple switches Unicast Flooding just happens.

HTH

Andy

 

[KiscoKid]

Hi

If you have a flat network with a single VLAN, then yes I'd expect you to see all traffic on any port where that VLAN is configured.

I personally would typically aim to put VOIP on its own VLAN.

I ran your error through the CCO, it came back with the following:

4. %SYS-4-P2_WARN: 1/Filtering Ethernet MAC address of value zero Problem
The switch is generating Filtering Ethernet MAC address of value zero messages. This is an example of the syslog output that you see when this error occurs: %SYS-4-P2_WARN: 1/Filtering Ethernet MAC address of value zero from agent host table interface %SYS-4-P2_WARN: 1/Filtering Ethernet MAC address of value zero from agent host table interface The Filtering Ethernet MAC address of value zero syslog message is generated when the switch receives packets with a source MAC address of 00-00-00-00-00-00, which is an invalid source MAC. The syslog message indicates that the switch refuses to learn the invalid address. However, the switch forwards traffic sourced from an all-zeros MAC address.

Recommended Action: The workaround is to try to identify the end station that is generating frames with an all-zeros source MAC address. Typically, such frames are transmitted from a traffic generator (for example, Spirent SmartBits), certain types of servers (such as load-balancing IBM WebSphere servers), a misconfigured router or end station (for example, a device transmitting all-zeros broadcasts), or a faulty NIC.

 

[JOAMON]

What is this telling me (port 2/44 is a span port for purposes of Websense)?

This looks like it was configured to see all traffic for Websense internet security product.

 

[jdeisenm]

How many pcs and how many phones on your network?

 

[hinesjrh]

jdeisenm asked:
How many pcs and how many phones on your network?

At this location (our corporate service center) there are ~150 employees who each have a PC and and an IP phone. There are also phones in conference rooms, etc. so I would say 150 PC's, 170 phones, and then there is our data center equipment. We have about 60 Windows servers that host applications that support our whole organization (this site and 50+ other sites). The IP phones are limited to this site - we are not running VOIP across our WAN to the 50+ sites.

Hopefully that gives you an idead of the nubmer of nodes here.

 

[jdeisenm]

Are your servers on the same network as the phones and pcs?
Do your phones support trunking?
What is websense watching? What are you spanning to that websense port?

I concure with previous posts. Segment your network and put your phones, pcs and servers on their own network. Also, you may be able to put a small switch or a tap in line and have the websense monitor computer plug into that.

 

[hinesjrh]

Are your servers on the same network as the phones and pcs?
Yes, some are and some are in our DMZ (pretty much a flat set up).

Do your phones support trunking? Yes, I believe they do.

What is websense watching? What are you spanning to that websense port? That span port on the Cisco 4006 is set up to watch all HTTP traffic.

 

[jdeisenm]

What are you spanning? Are you spanning the vlan or a specific port?

 

[ADB100]

I still believe this is likely to be unicast flooding. Please verify by making a note of the destination MAC addresses from your sniffer trace. Telnet to the switch and check whether the MAC addresses have been learned:

!IOS
show mac-address table address 1234.1234.1234

!CatOS
show cam 12-34-12-34-12-34

If the MAC addresses are not in the CAM table then the switch will simply flood the traffic out all ports in the VLAN.

HTH

Andy

 

[hinesjrh]

jdeisenm (MIS) asked:
What are you spanning? Are you spanning the vlan or a specific port?

Here's the output regarding my span port on the 4006 switch. I am spanning my whole primary vlan. The first thing I see is all the Xmit errors on that span port. I have to see why that is happening! Don't you agree?

Console> (enable) show span all

Destination : Port 4/20
Admin Source : VLAN 2
Oper Source : Port 2/4,2/9,2/12,2/19-21,2/25-28,2/30-41,2/43-48,3/29,3/35-36
,3/47,4/1-3,4/7-9,4/13-14,4/16-18,4/21,4/26-27,4/29-34,4/36-38,4/40,4/43-48,5/1-
9,5/20,5/22-23,5/25-26,5/30,5/32,5/34,5/36,5/38,5/40,5/46
Direction : transmit/receive
Incoming Packets: disabled
Learning : enabled
Filter : -
Status : active

------------------------------------------------------------------------
Total local span sessions: 1

No remote span session configured
Console> (enable)

Console> (enable) show port 4/20
* = Configured MAC Address

Port Name Status Vlan Level Duplex Speed Type
----- ------------------ ---------- ---------- ------ ------ ----- ------------
4/20 monitor 1 normal a-full a-100 10/100BaseTX

Port AuxiliaryVlan AuxVlan-Status InlinePowered PowerAllocated
Admin Oper Detected mWatt mA @51V
----- ------------- -------------- ----- ------ -------- ----- --------
4/20 none none - - - - -


Port Security Violation Shutdown-Time Age-Time Max-Addr Trap IfIndex
----- -------- --------- ------------- -------- -------- -------- -------
4/20 disabled shutdown 0 0 1 disabled 134

Port Num-Addr Secure-Src-Addr Age-Left Last-Src-Addr Shutdown/Time-Left
----- -------- ----------------- -------- ----------------- ------------------
4/20 0 - - - - -

Port Flooding on Address Limit
----- -------------------------
4/20 Enabled

Port Status Channel Admin Ch
Mode Group Id
----- ---------- -------------------- ----- -----
4/20 monitor auto silent 128 0

Port Status ErrDisable Reason Port ErrDisableTimeout Action on Timeout
---- ---------- ------------------- ---------------------- -----------------
4/20 monitor - Enable No Change

Port Align-Err FCS-Err Xmit-Err Rcv-Err UnderSize
----- ---------- ---------- ---------- ---------- ---------
4/20 - 0 6304963 0 0

Port Single-Col Multi-Coll Late-Coll Excess-Col Carri-Sen Runts Giants
----- ---------- ---------- ---------- ---------- --------- --------- ---------
4/20 0 0 0 0 1 0 0

Last-Time-Cleared
--------------------------
Wed Apr 26 2006, 23:26:35

Idle Detection
--------------
--
Console> (enable)

 

[jdeisenm]

Do you need to monitor what happens on the whole vlan or only "uplink" port to the internet? You can change your span to only monitor that uplink port and websense will still see all your internet traffic. That would drastically cut the work the switch does with that span session and websense would still see the outbound http traffic.

 

[hinesjrh]

No, I have changed it to look at a lot less! Thanks.

Destination : Port 4/20
Admin Source : Port 2/4
Oper Source : Port 2/4

However, I am still getting the warning message:

Console> (enable) show cam count dynamic
2006 May 04 15:34:50 %SYS-4-P2_WARN: 1/Filtering MAC address 00-00-00-00-00-00 on port 2/44 from host table
Total Matching CAM Entries = 508

Port 2/44 on my 4006 switch relates to my Websense server. I know on that server there is one NIC in promiscous mode and it does not have an IP address assigned to it. Would that make it send out from an all zero MAC address?

Also when running ethereal from any port in our office I am seeing network traffic from all over our network (including remote locations on the otgher end of our WAN links).

 

[jdeisenm]

What is the configuraton on your websense monitor nic?
What does Ipconfig/all say the mac address is on that nic?

Our websense server has 2 nics. The monitor nic is plugged in to a monitor port on a 4507 running IOS. The monitor nic has all the "bindings" unchecked except for Network Monitor Driver. The advanced driver properties > locally administered address is set to "not present".

 

[hinesjrh]

We finally found what was causing all our network traffic. It seems that our Microsoft Network Load Balanced Cluster servers running all our webs are creating a lot of unicast traffic. It seems that when you have a clustered scenario the clustered NIC shares an virtual MAC address between the two servers. My Cisco 4006 switch w/ Sup II (layer 2, not 3) can't tell what MAC relates to what server so it doesn't keep it in the table. Since the MAC is not in the table it sends these unicast packets all over my network.

Once I replace my 4006 with a new layer 3 switch and set up appropriate VLAN's for the clustered servers I should rid myself of this headache! Any suggestions for a replacement for my 4006 are welcome.

 

[NetworksMD]

Could you not put a static cam statement on each of the interfaces if this is causing your issue?

 

[paulk29]

Cisco 4507R is a very good L3 Switch.

Paul Kilcoyne B eng. CCNA