Cat 4006 w/4232 L3 blade loses connection btw Sup @ 4232

[marvandkelly]

I've got an issue where I have multiple Cat 4006 switches with 4232 blades. The switches run for months with no issue. Then the 4232 seems to lose connectivity with the sup via the back-plane.

Our set-up is standardized as such in each of our distribution closets and server farm:

2 4006 Switches, each with the 4232 Layer-3 blade and Sup2's. (4006A and 4006B)
The each 4232 has one gig-link back to our core pair of 6509's.
The 4006's are connected via L2 through the gig port of the supervisors, so 4006A connects to 4006B via Sup1 port G1.

On the 4232 we have 5 to 6 port-channels set-up that hosts the L3 subnets on the 4006's, and we have corresponding VLAN's on the Sups for each port-channel.

When the 4232 loses connectivity to the sup (say on 4006A), I can telnet to 4006B then to the sup on 4006A. From there I should be able to session to the 4232, but am not able to.
I can also telnet to 4006A's 4232 via the L3 gig-link to our 6509's. But a sh cdp nei shows that the blade no-longer sees the supervisor.

We've opened a multitude of cases with TAC and worked with our SE to resolve this issue, but Cisco seems to be at a loss as to why it's happening.

Has anyone else experienced simular situations?

 

[rburke]

I'm not sure if you have any extra slots, but you might want ot try moving the blade to another slot. If it has been working and then just suddently started happeneing then that usually points to a hardware problem. What kind of troubleshooting steps have you taken already?? Does the TAC have any ideas on what is happeneing?? Is this happening on both 4006 A and B or just A??

Burke

 

[marvandkelly]

) TAC has suggested moving the 4232 to another slot. We're going to schedule some down time to do this since they're all in a production enviroment.
They're also asking us to move the native VLAN to something other than VLAN1, allthough this goes against all published documentation that Cisco puts out.

Troubleshooting wise, we have tried pings from the failed 4232 to the sup. Adjusting packet sizes. One strange thing, in some of the switches, it appears that packets >1472 bytes are able to pass. TAC has had us attempt pings to multicast addys (ie. EIGRP's multicast address). They've even had developers in out routers looking around.

It does happen on the "A" and "B" switch, but thankfully not on the "A" and "B" at the same time. We have 12 pairs of 4006's running in our campus, right now 4 of them are limping along.

Cisco has identified a possible hardware issue with the 4006's, 4232's and having VLAN1 as native vlan.

 

[dmathews]

This is happening on one of my Cisco 4006s right now. What was the fix? Cisco is sending us new CAT OS for the Supervisor II, but they don't sound too sure that it will fix it.

 

[evRouterJockey]

This is the nuts and bolts of the case:

There are several of these switches w/L3 cards (4232) and Sup2 Engines in the network. All of the remote sites, all running cat4000.6-4-6.bin OS and IOS Version 12.0(10)W5(18f), with various HW versions, ranging from 1.5 to 1.7 are all running normally. The switch at the core meets all of the same criteria, yet is experiencing the loss of connectivity to the L3 card. It stops routing, you can't ping any of the IP addresses on the card, you can't session to it, and you can't console to it. We've logged messages to a syslog server and nothing. We've put a PC on the console port of the L3 card and logged messages to Hyperterm, and nothing. We've even built a completely new switch, blade for blade, using a different chassis, sup module, L3 card, and have now experienced the same exact thing.

Cisco TAC case is out there for anyone to review and comment on: E869020.

 

[dx1]

We had the same problem awhile back. Got new sup and new L3 card and still same problem. Had to replace the whole chassis and all the modules.

 

[dmathews]

DX1,

Thanks for the reply. Did anyone mention anything about IOS/CAT OS upgrades or age of the modules? I have spares of everything, but they are the same age as the failing chasis and cards and I'd like to know if I'm going to see the same issue on my replacement parts.