Can't ping switches

[AspenMan]

On occasion I have three switches in the same location that I can not ping or access from multiple Vlans\PCs. Users on those switches continue to have access to our network (internet, data drives). We use InterMapper to monitor our switches and it shows the switches as down. However if I log into a different switch I can ping the down switches. Once I ping them from inside that switch I can then ping the down switches from a Windows command prompt and InterMapper shows the the switches as Up.

Can anyone explain this behavior?

 

[VinceWhirlwind]

What subnet are the switches on?
What subnet is intermapper on?
What default routes do the switches have on them?

 

[AspenMan]

Whirl -

All of our switches are on 192.168.254.xxx subnet, Vlan 1 Management LAN. Intermapper is on vlan 430. All vlans can talk to each other and are routed by our 4510 or 6509. Pinging and access are the same from any Vlan whether the switches are showing up or down. The problem with the switches mentioned is an intermittent problem and only on these switches out of 60 or 70 on the network.

 

[VinceWhirlwind]

Hang on, you've contradicted yourself:
You said initially:

"InterMapper ...shows the switches as down. However if I log into a different switch I can ping the down switches."

Now you say:
"Pinging and access are the same from any Vlan whether the switches are showing up or down"

Your initial scenario indicated that the switches with the problem have something like a conflict in their routing table. A sh ip rout might reveal what this is.

I'm interested that you say your inter-vlan routing is done by:
"our 4510 or 6509"

How does that work?

 

[AspenMan]

Correct.

All our vlans are setup on our 4510 and 6509 routers. We allow all vlans to talk to each other. the switches have no routing on them.

If the switches in question are down I can access them no matter what vlan I use, I guess except the management vlan (vlan 1).

If the switches in question are up I can access them from any vlan.

The weird thing is that once I ping the swiches in question via Vlan 1 (ie from another switch) they immediately become available again from all vlans. These switches have been in place for years with no issues and as far as know nothing has changed on the switch configurations. However something has changed, I just don't know what.

 

[VinceWhirlwind]

There is still something not quite right about the info you are giving:

"InterMapper ...shows the switches as down. However if I log into a different switch I can ping the down switches."

...contradicts:

"If the switches in question are down I can access them no matter what vlan I use, I guess except the management vlan (vlan 1). "

So, (assuming "sh ip rout" doesn't explain anything) what is the default GW on those switches?
Where is that address on the network?
What addresses do your two routers have for each VLAN? (They can't both be the default GW for the VLAN, can they?)

Anyway, if you can't figure it out logically, you can document the problem using packet captures: create a mirror port on the dodgy switch, wait until the next time they go "down", then capture the traffic coming from Intermapper to the switch, you will see the ICMP packets (if that's what they are) coming into the switch, and no replies. Or mis-addressed replies. Or unexpected ARP requests.
You should check your mac-address table on the switch against the mac address in the ICMP packets. You should note any other traffic leaving the dodgy switch.

 

[AspenMan]

You are correct. This is a typing error. My statmeent should have read.

"If the switches in question are down I can't access them no matter what vlan I use, I guess except the management vlan (vlan 1).

 

[AspenMan]

The routers are set so one is the primary and the other is secondary for VLAN control. If one router dies the other takes over vlan routing on our data rings.

Here is the rest of the data. sh ip rout is not a legal command on these switches (Cisco 35xx - to be replaced next year). I will look into setting up a mirror port.

plaza1sw2#sh ip arp
Protocol Address Age (min) Hardware Addr Type Interface
Internet 192.168.254.254 176 0000.0c07.ac01 ARPA VLAN1
Internet 192.168.254.23 201 0002.7d7c.b580 ARPA VLAN1
Internet 192.168.254.24 - 0002.7d7c.ea80 ARPA VLAN1
Internet 192.168.254.105 122 0003.6b62.f000 ARPA VLAN1

plaza1sw2#sh ip rout
^
% Invalid input detected at '^' marker.


plaza1sw2#sh run
Building configuration...

Current configuration:
!
version 12.0
no service pad
service timestamps debug uptime
service timestamps log datetime
no service password-encryption
service sequence-numbers
!
hostname plaza1sw2
!
enable secret 5 $1$iAwy$20738yajtThYNI0bOP0Hv.
enable password 7 065F562E5C4B07
!
!
*******************
!
interface VLAN1
ip address 192.168.254.24 255.255.255.0
no ip directed-broadcast
ip nat outside
no ip route-cache
!
ip default-gateway 192.168.254.254
logging 10.4.20.56
snmp-server engineID local 00000009020000027D7CF300
snmp-server community public RO
snmp-server community private RW

 

[VinceWhirlwind]

I suppose your routers use HSRP - if that were not working, you would have more than a couple of switches complaining....

It doesn't look like a layer3 issue - your switches have a simple config, no ip routing, good default GW.

When the switch is "down", compare the source/dest mac addresses on the ICMP packets with the ARP and mac-address tables.