VLAN1 will not take an IP address

[ksas025]

I have a 2950 24 port switch and I am trying to assign an IP to vlan1 for management purposes. To do so I did the follwoing:

config t
interface vlan 1
ip address 10.24.64.5 255.255.255.0
no shut

When I attempt to ping something on the local LAN (10.24.64.110 for example) it fails. When I issue a sh ip arp command it shows the IP address I just attempted to ping with a hardware address of Incomplete.

Why wont my Switch communicate on the local LAN?. My config is as follows:

Current configuration : 5414 bytes
!
! No configuration change since last restart
!
version 12.1
no service pad
service timestamps debug uptime
service timestamps log uptime
service password-encryption
!
hostname swA
!
no logging console
enable secret xxxxxxxx
enable password 7 xxxxxxxx
!
clock timezone CST -6
clock summer-time CST recurring 1 Sun Apr 1:00 last Sun Oct 2:00
ip subnet-zero
no ip finger
no ip domain-lookup
!
!
no spanning-tree vlan 1
no spanning-tree vlan 2
no spanning-tree vlan 3
no spanning-tree vlan 4
!
!
interface Port-channel1
 switchport mode trunk
 flowcontrol send off
!
interface FastEthernet0/1
 description VLAN1 - DMZ Lan Segment A - 10.24.64.0 - Port 1
 duplex full
 speed 100
 spanning-tree portfast
!
interface FastEthernet0/2
 description VLAN1 - DMZ Lan Segment A - 10.24.64.0 - Port 2
 duplex full
 speed 100
 spanning-tree portfast
!
interface FastEthernet0/3
 description VLAN1 - DMZ Lan Segment A - 10.24.64.0 - Port 3
 duplex full
 speed 100
 spanning-tree portfast
!

.............
...Edited for space
.............


!
interface FastEthernet0/20

 switchport access vlan 4
 duplex full
 speed 100
 spanning-tree portfast
!
interface FastEthernet0/21

 switchport access vlan 4
 duplex full
 speed 100
 spanning-tree portfast
!
interface FastEthernet0/22

 switchport access vlan 4
 duplex full
 speed 100
 spanning-tree portfast
!
interface FastEthernet0/23
 description Trunk to other switch - Port 23
 switchport mode trunk
 channel-group 1 mode on
!
interface FastEthernet0/24
 description Trunk to other switch - Port 24
 switchport mode trunk
 channel-group 1 mode on
!
interface GigabitEthernet0/1
!
interface GigabitEthernet0/2
!
interface Vlan1
 description VLAN1 Primary Lan Segment (10.24.64.0).
 ip address 10.24.64.5 255.255.255.0
 no ip route-cache
!
interface Vlan2
 no ip address
 no ip route-cache
 shutdown
!
interface Vlan3
 no ip address
 no ip route-cache
 shutdown
!
interface Vlan4
 ip address 10.24.11.6 255.255.255.0
 no ip route-cache
 shutdown
!
ip http server
logging trap debugging
logging facility local5
logging 10.24.99.75
access-list 20 permit 10.24.11.0 0.0.0.255
access-list 20 permit 10.24.12.0 0.0.0.255
access-list 20 permit 10.24.13.0 0.0.0.255
access-list 20 permit 10.24.14.0 0.0.0.255
access-list 20 permit 10.24.15.0 0.0.0.255
access-list 20 permit 161.134.120.0 0.0.0.255

!
line con 0
 exec-timeout 0 0
 transport input none
 stopbits 1
line vty 0 4
 access-class 20 in
 password 7 xxxx
 login
line vty 5 15
 access-class 20 in
 password 7 xxxxx
 login
!

end

I also captured traffic on the local LAN and found that the switch was sending out an arp request but never seeing a reply. Since its on the local (layer 2) lan I would think the default gateway is irrelevant. Furthermore, the destination node has the correct Arp entry in its local table for the switch. It seems that the switch is filtering or unwilling to processes arp requests from the local lan. Any Ideas?

 

[ksas025]

Thanks voltron but I entered the default gw (which happens to be a PIX firewall on this segment)and still no dice.

A little off subject......

I opened a TAC request with Cisco and one of their recommendations is I point the default-gateway of all of my hosts to the switch's IP address (or the IP address I am attempting to configure). Then the default gateway configured on the switch will forward the appropriate packets to the true segment gateway at the 64.1 address. Does this make sense to anybody? It might make sense to me if my switch were a layer 3 device but its not (if it is I am not using it as such). Anyways just though Id bounce that off of you guys.

Continuing......

I have some new information. I turned on ARP debugging on the switch and its a little interesting. It shows ARP requests being sent out for the default-gw at 10.24.64.1 with destination MAC 0000.0000.0000 which I believe is a broadcast address. Then, the next field of the debug line indicates the interface being used is vlan2. I would suspect that the VLAN associated with my ARP requests be vlan1 since that is the SVI that is not shutdown and all my switchports are on vlan1.

I dont know what to say. The switch is just misbehaving. It should not be using VLAN 2 to send ARP requests to assets on VLAN1.

The TAC represetative also said that a posible reason for my problem might be the firewall on my segment. I dont see how. Does anybody?

Thanks for everyones patience, persistence, and willingness to help in what seems to be a hopeless problem.

Alex

 

[helpdeskdan]

Yes, a switch will not forward traffic in between vlan1 and vlan2 if it is not a layer 3 switch. The two are separate as if they were different switches. It must be the firewall doing so.

Are you using span for wireshark? If not, you will not get the replies with wireshark because those are unicast, not broadcast.

 

[globalchicken]

1 thing that i noticed was the no spanning-tree on vlan 1. This could induce a serious broadcast storm on your network. I saw that you had at one time a etherchannel on ports 23/24 trunking to another switch, but you did say nothing was connected anymore? I would turn spanning-tree back on.

MC

 

[ksas025]

Helpdeskdan: Yes, I did use a span port for my capture so I did catch the unicast ARP replies as well as the Ping replies that the switch seemed to ignore. I captured a telnet from the switch to a UNIX box which is physically connected to the switch and on a port assigned to vlan1. The capture showed SYN,ACKs comming back from the Unix host to the switch. I did not see the SYN, I assume because it is comming from the SVI which is probably not included in the Span session? Anyways, the SYN,ACKs are ignored by the switch. I sent the TAC team my findings in the form of a screenshot; eveidence that cannot be ignored. Can I post pics on here?

gchicken: Spanning tree was turned off before I arrived at this job and since I am un-clear of why it was turned off I think I am gonig to leave as is. Everything works fine. Except my VLAN IP problem of course.

 

[aletoledo]

Since you appear to be able to change a lot of the configuration of this switch, can you change the trunck port you have on "interface FastEthernet0/24". either turn it off/unplug it if it isn't being used by anything else.

Maybe your arp packet is traversing the trunk somehow and that is causing the packet to somehow get tagged to vlan2? maybe try adding the command to both switches on each side of the trunk: "dot1q native vlan 1

 

[helpdeskdan]

dot1q native vlan 1 is default, but who knows - it's acting strange. Now the race is on; can we solve the problem before Cisco TAC? That would a feather in our cap indeed.

Can you disconnect the firewall from the network and see if it behaves correctly? That would at least assure the firewall was not doing something strange. If you wish to post pictures, just upload them and post a link; we'll look at it.

 

[ksas025]

This is a production switch; I cannot disconnect the firewall from it.

Im glad you taking a competitive attitude towards it. Given my recent conversations with TAC, I put my money on you guys/gals. TAC is at an advantage right hough, I just sent them my Layer-2 topology about 3 hours ago.

I would like to upload a picture but Im not sure how. When you say "just upload" where do you mean?

 

[helpdeskdan]

Post it on a webpage somewhere & give use the URL. Maybe the configs too, minus passwords and first octet of public addresses.

 

[aletoledo]

I can't imagine how the toplogy will help out any much more. I think the problem appears to be fairly well defined now.


1) the switch is sending out a valid arp.
2) the host sees the arp and replies correctly, but the switch fails to record the mac.
3) the switch shows arps only on VLAN 2.

I honestly think its a tagging issue. However its strange that the initial arp is seen by the host, yet is being sent on VLAN 2? I don't think the switch pings should be flooded across all VLANs, but should remain on the correct VLAN.

Wireshark does capture VLAN tags, so I wonder if the packet leaves untagged and then traverses the tunnels, ends up somehow tagged as VLAN2 and gets lost then.

I looked up "dot1q native vlan 1" and it says that there isn't any default value (perhaps thats only 6500 series though).

 

[ksas025]

Sorry, nowhere to post it. But maybe some ascii art will do the trick!


Pix Failover Link
-----------------------------------------
| |
| |
------ ------ ------ ------
| |-----| |----| |-----| |
|_P1_| |_S1_| |_S2_| |_P2_|
| |
| |
Layer 2 net Layer 2 net

Sorry for the picture but it gives sufficient info I think.
All connections between devices are Layer 2 links. The PIX's
are in Active Failover mode which means only one PIX owns the
virtual default-gw address of .1 at any given time. Since
the etherchannel between the switches is down (at least I think it is)
I believe the link between the switches is simply a layer 2 uplink for vlan1.
The hosts that I attempt to ping from are physically hanging off
of S1.

All hosts on the Layer two networks have active backup NICs which allows them to be connected to both switches but only have one NIC active at a time. The MAC address on the NIC is shared between both switches.

I hope this helps! And thanks to everyone who has contributed to this thread.

 

[ksas025]

For anyone who is interested, TAC finally suggested that I update my IOS due to a bug listed here:

http://www.cisco.com/cgi-bin/Support/Bugtool/onebug.pl?bugid=CSCdx26212&Submit=Search

I updated my IOS and it started working immediately.

Thanks again for everyones input and suggestions.

 

[ksas025]

Sorry about the link. it requires a password. here is the bug:

Problem:
Once the switch has been up and running for a while
it looses ip management where the user can no longer ping, telnet, etc.
Users continue to pass traffic during this time and doesn't show to be
affected.


Analysis:

The bug was in the handling of 802.3 SNAP encapsulated frames
(DSAP is 0xAA and SSAP is 0xAA) on the management VLAN with
the following destination address in the destination address
field:

Broadcast (FFFF.FFFF.FFFF)
Switch MAC address

These packets are received by the CPU. There was a buffer
leak for 802.3 SNAP encapsulated frames if the type field in
the SNAP header is NOT of the listed type given below. Over
time this buffer leak depleted the buffer pool; after the CPU
runs out of buffers, packets to CPU are dropped.

Problem frame would be any frame with the following format that
has a type field not equal to the list given below (Field
Lengths are given in parenthesis):

________________________________________________________________
|FFFF.FFFF.FFFF or |Source |Length|0xAA|0xAA|0x03|OUI|Type| |
|switch MAC Address|Address| | | | | | |Data|
| (6) | (6) | (2) | (1)| (1)| (1)|(3)| (2)| |
----------------------------------------------------------------

The fix is to release the buffer when a packet is received by
the CPU that is not of the listed type given below.

0x0800 - IP
0x0BAD - VINES
0x80C4 - VINES2
0x8137 - NOVELL1
0x809B - Ethernet Apple Talk
0x0806 - ARP
0x8035 - RARP
0x1996 - RSRB
0x80F3 - AppleTalk ARP
0x0105 - Layer 3 switching on earl 2
0x0107 - Multicast Shortcut Control protocol
0x010F - Catalyst Serial Communication protocol
0x0113 - Forwarding Table Edit protocol

0x0112 - *Cluster Management protocol
0x0114 - *Cluster RARP
0x0115 - *Cluster Layer 2 protocol

* The OUI field in the SNAP header should be 0x00000c.

From the list of packet types given above the following types
are passed to the protocols and the rest are dropped:

0x0800 - IP
0x0806 - ARP
0x8035 - RARP
0x0112 - Cluster Management protocol
0x0114 - Cluster RARP
0x0115 - Cluster Layer 2 protocol


Workaround:
The only way to recover is to reload the switch.


Fix:
The fix will be available in an interim release due out 7/15/2002 (image name
unknown) as well as 12.1(11)EA1.