VLAN database disappeared. Why? 1

[eNur]

Recently I got 'bitten' during a supposedly common configuration/setup with our Cisco Catalyst C2950. We fixed the problem but did not know what's going on, so we don't know how to avoid it in the future.

Background Story:
Cisco Catalyst C2950 running with some VLANs already set, VTP as Server, Trunk to another C2950. Everything running as it should (VLAN and all).
Planned to connect a Trunk line to Cisco Catalyst Express 500.
Cable connection between CE500 to C2950 is set, CE500 port was put as SwitchUplink (to accept dot1q Trunk). C2950 port is still at "switchport mode access", so Trunking is not running yet. All LED are green.

The problem:
At C2950, for the specified port, I typed "switchport mode trunk".
Some seconds later (maybe 10-30 seconds?) all ports went amber, and all connections went dead. Thinking that I caused the problem, I tried to reverse what I did "switchport mode access".
After waiting a long time, the LEDs are still amber. Because this is a production Switch and we need active connection ASAP, we disconnected the connection between C2950 and CE500, and restart the C2950.
After restart, the LEDs are still amber.
Checking "show running-config" and comparing with the documented configuration showed that everything is exactly the same. However, checking "show vlan" showed that all ports were not assigned to any VLAN.

The emergency fix:
Let's say I am on port 3, which supposed to be in VLAN 2.
The "show running-config" for port fa0/3 showed that it is indeed "switchport mode access" and "switchport access vlan 2".
But "show VLAN" shows that there is no "vlan 2"
So, I typed "switchport access vlan 2" (just copying what already there in "show running-config") and the Console returned with "Access VLAN does not exists. Creating vlan 2". With this done, the LED on port 3 (and others belonging to VLAN 2) changed to green.
Repeated for the rest of unregistered VLAN fixed the problem, everything run ok as before.

My question:
Does anyone know why the VLAN database went missing?
We discussed this with our local 'expert', and he also as stumped as everyone else.

Thank you for any insight to the problem.

 

[burtsbees]

You trunked to a switch that was in VTP Server mode that rewrote the vlan database with its own config, which was blank, because of the higher vtp revision. When connecting new switches, ALWAYS make sure they are in Transparent mode, as most new switches default to be in VTP Server mode.
Evil prank by Cisco...

Burt

 

[eNur]

Ah, I missed this one.
Need to update the Operating Procedure to make sure that all newer Switches will be VTP Transparent from now on, to avoid future problems.

However (fortunately?) after I checked the docs, all Cisco Catalyst Express 500 series switches support only VTP transparent mode, thus they can't be set to be VTP Server or Client. So, we can rule out this as the possible cause of the problem.

Thank you for the heads up though, because my group will set up a similar installation using Catalyst 3960. We will make sure that the VTP mode will be Transparent, so we can avoid the Cisco prank.

Any other ideas?

My colleagues suggested that after 4 years running, the C2950 was due for memory corruption which killed the VLAN database. I doubt it because it is running fine now, but is this possible?

 

[burtsbees]

post a sh log and sh mem...well, sh log anyway, as sh mem would be long...

Burt

 

[jneiberger]

I highly doubt that the CE 500s only support transparent mode. That wouldn't make sense for Cisco to introduce a switch that can't receive VLAN information from the rest of the network.

Can you provide us with the URL of the documentation that says that?

 

[jneiberger]

Okay, nevermind. I found the documentation myself and he's right. The CE500 only supports transparent mode.

Yet another reason not to ever get one. I've heard nothing but bad things about the CE500 switches. I was even told once by an engineer at Cisco not to buy them. I don't think the Cisco employees like them at all.

 

[burtsbees]

Well, with no command line? Screw that! I did read in a few posts that there is a way to get a command line, but there's hacking the fl;ash involved...
Now as to the problem in this post---it really looks like to me the trunk sent some updated info...

Burt

 

[vipergg]

Your assumption sounds correct about the vlan database , it somehow got corrupted or wiped. any possibility of having a loop from the cat500 to the 2950 ? Once the vlan database got corrupted the ports went amber because all those ports were put into a "inactive" state because they were in a vlan that was not active . You can see this by doing a "show interface status" command when it happened. If you only have a couple of small switches in your domain then transparent is a better option, client server is better utilized in larger installs . It works fine as long as you understand VTP and the config revision levels and the ramifications. Whenever installing a new switches , wipe the current config out , change the vtp domain name to something other than what you are using then set it back and that will reset the config revision to "0" and it can't overwrite the vtp database. Why it happened in your case i really don't know , to me it sounds like it got some kind of spanning loop introduced somehow and it corrupted the database , 30-50 seconds is the time it takes for spanning tree to run .