Flapping Stack Link on 3750

[Mankyway]

This Morning I awoke to find that the following message was being generated at around 4000 MPH

Anyone any idea how to resolve it? The cables have been checked and traffic is not affected.

Any other information required, please ask!!

I did note that the software image on the "master" is older than the slave, I recall this being a Gotcha on Nortel kit, I was wondering if the same could be said of Cisco as well?

Thanks in advance

2008-12-05 14:08:19 Local7.Info 18316: *Mar 1 02:10:12: %STACKMGR-6-STACK_LINK_CHANGE: Stack Port 1 Switch 2 has changed to state DOWN

2008-12-05 14:08:19 Local7.Info 18317: *Mar 1 02:10:13: %STACKMGR-6-STACK_LINK_CHANGE: Stack Port 1 Switch 2 has changed to state UP

2008-12-05 14:08:19 Local7.Info 18317: *Mar 1 02:10:13: %STACKMGR-6-STACK_LINK_CHANGE: Stack Port 1 Switch 2 has changed to state UP

2008-12-05 14:08:21 Local7.Info 18318: *Mar 1 02:10:14: %STACKMGR-6-STACK_LINK_CHANGE: Stack Port 1 Switch 2 has changed to state DOWN

2008-12-05 14:08:21 Local7.Info 18318: *Mar 1 02:10:14: %STACKMGR-6-STACK_LINK_CHANGE: Stack Port 1 Switch 2 has changed to state DOWN

2008-12-05 14:08:21 Local7.Info 18319: *Mar 1 02:10:15: %STACKMGR-6-STACK_LINK_CHANGE: Stack Port 1 Switch 2 has changed to state UP

2008-12-05 14:08:21 Local7.Info 18319: *Mar 1 02:10:15: %STACKMGR-6-STACK_LINK_CHANGE: Stack Port 1 Switch 2 has changed to state UP

2008-12-05 14:08:23 Local7.Info 18320: *Mar 1 02:10:16: %STACKMGR-6-STACK_LINK_CHANGE: Stack Port 1 Switch 2 has changed to state DOWN

2008-12-05 14:08:23 Local7.Info 18320: *Mar 1 02:10:16: %STACKMGR-6-STACK_LINK_CHANGE: Stack Port 1 Switch 2 has changed to state DOWN

2008-12-05 14:08:23 Local7.Info 18321: *Mar 1 02:10:17: %STACKMGR-6-STACK_LINK_CHANGE: Stack Port 2 Switch 1 has changed to state DOWN

2008-12-05 14:08:23 Local7.Info 18321: *Mar 1 02:10:17: %STACKMGR-6-STACK_LINK_CHANGE: Stack Port 2 Switch 1 has changed to state DOWN







3750#sh switch detail

Current

Switch# Role Mac Address Priority State

--------------------------------------------------------

1 Slave 0012.7f1b.xxxx 1 Ready

*2 Master 0011.bbc0.xxxx 1 Ready







Stack Port Status Neighbors

Switch# Port 1 Port 2 Port 1 Port 2

--------------------------------------------------------

1 Ok Ok 2 2

2 Ok Ok 1 None



3750#show switch 1 ?

| Output modifiers

<cr>



3750#show switch 1

Current

Switch# Role Mac Address Priority State

--------------------------------------------------------

1 Slave 0012.7f1b.xxxx 1 Ready







3750#SHOW SWITCH DETAIL

Current

Switch# Role Mac Address Priority State

--------------------------------------------------------

1 Slave 0012.7f1b.xxxx 1 Ready

*2 Master 0011.bbc0.xxxx 1 Ready







Stack Port Status Neighbors

Switch# Port 1 Port 2 Port 1 Port 2

--------------------------------------------------------

1 Ok Ok 2 2

2 Ok Ok 1 1



3750#show switch neighbors

Switch # Port 1 Port 2

-------- ------ ------

1 None None

2 None None



3750#show switch stack-ports

Switch # Port 1 Port 2

-------- ------ ------

1 Ok Ok

2 Down Ok



3750#show switch stack-ports

Switch # Port 1 Port 2

-------- ------ ------

1 Ok Ok

2 Ok Ok



3750#show switch stack-ports

Switch # Port 1 Port 2

-------- ------ ------

1 Ok Ok

2 Down Ok







Switch Ports Model SW Version SW Image

------ ----- ----- ---------- ----------

1 26 WS-C3750-24TS 12.1(19)EA1d C3750-I9-M

* 2 12 WS-C3750G-12S 12.1(19)EA1d C3750-I5-M





Switch 01

---------

Switch Uptime : 6 hours, 5 minutes

Base ethernet MAC Address : 00:12:7F:1B:xx:xx

Motherboard assembly number : 73-xxxx

Power supply part number : 341-xxxx

Motherboard serial number : C

Power supply serial number : PHIc

Model revision number : F0

Motherboard revision number : A0

Model number : WS-C3750-24TS-S

System serial number : CAT



Configuration register is 0xF

 

[vipergg]

That is probably a hardware issue , seems to me I remember reading a tsb about faulty stack cables its possible you are running into something like that if you are sure both ends are tightly screwed down . I have found that they can look like they are in tight but they aren't and you have to make sure the thumbscrews are tight do to flimsy nature of the plugs.

http://www.cisco.com/en/US/docs/swi...re/troubleshooting/switch_stacks.html#wp58952

 

[PeterHurst]

We have a similar problem - we are still in process of diagnosing but suspect a faulty switch stack cable.

I'll try and repost later in the week when my Engineer gets to visit.

 

[Mankyway]

Many thanks for your responses, reading up on the problem, it appears to be leaning toward a faulty cable(s).

Peter, I would be very interested in the resolution of your problem.

 

[PeterHurst]

We have done some remote diags - show platform stack-manager counters - which wasn't too helpful but showed 500+ stack changes whereas a normal stack had around 3.

Studying the counters on the UP/DOWN and IN/OUT it appeared that one direction was around double the other - whilst on the good stack they were reasonably even.

Nothing was clear so we disconnected the cable that hasn't been replaced and the stack speeded up.

Later in the week we indent to change that cable and possibly change the switch the user is on.

 

[VinceWhirlwind]

I've had a faulty stackport.
A switch in a stack was performing badly - when I tried consoling into it, I got the message "insufficient memory".

When I looked at the back of it, I saw that the guys who installed the switches had dropped one and I could see quite clearly that the stackport was bent.
Also, they hadn't bothered connecting stack cables in both directions.

When I added a stack cable (to close the loop), the problem stopped.

 

[LincolnIT]

This may or may not be related to your issue, but you mentioned that the IOS images were different versions etween your Master and Slave switches. Cisco recommends that the images on all stack members are the same. Different versions, even a minor version, can result in unexpected behavior.

 

[Mankyway]

Many Thanks for al your help and advice we have connected the two switches via Fa ports and this has stopped the problem until we get a new switch cable

 

[JMoore64]

Interesting, I am experiencing the same issue on a stack of four 3750-48PS swithes. we just updated the stack to IOS 12.2(46)SE. I didn't notice the issue prior to the update, So i'm not sure it's the IOS. It only happens on Switch 1 and Switch 2. I ran this command and caught it stating the Ring speed was 16G - Half.

3750-48# sh switch stack-ring speed

Stack Ring Speed : 16G
Stack Ring Configuration: half
Stack Ring Protocol : StackWise

Just thought i would drop my experence in here too. I will try and replace the Cable as soon as i can get some downtime approved.

Is there a command to see if there are any errors on the Stack ports?

 

[tjbradford]

if the 3750's are connected via a stack the master should apply the ios to all the other switches, automatically. so it could again point to a faulty stack cable.

 

[JMoore64]

Well i was able to get some downtime approved and after completing some other work i swap out the stack cable going from switch-1 port-1 to switch-2 port-2.... Ah shoot, i was just about to say so far so good, but then this just poped up:

2w4d: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 2 has changed to state DOWN
2w4d: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 2 has changed to state UP

Well, Changing the stacking cable didn't work...meh..

 

[burtsbees]

Seems those cables plug into a socket that may jar some solder joints loose. Isn't there a lifetime warranty on those switches?

Burt

 

[shoeman09]

Hello all,

I have had this issue for a few years now. In every case I have had to RMA the 3750 because the failure was hardware related and not the stackport cable. I have replaced at least 5 3750's and finally requested a failure analysis on the RMA, FA-0050863. Actually getting that was like pulling teeth! Don't get me wrong, I love Cisco but something just doesn't seem right with this issue.
When I first started calling TAC about this I had to do all sorts of trouble shooting with them. Now once I explain the problem they just start the RMA. The two I sent in for the FA surprisingly would not boot when they received them.

Now yesterday I have found 2 more switches with stack port failures...

I attended Cisco Live in Orlando last year and met someone having these same issues. They thought they had narrowed it down to heat issues. I looked into that and added fans to the enclosures and made sure the AC was cooling well in the closets. Looked good for awhile until yesterday.

By the way, the switches showing the bad stack ports...I have already replaced the sames ones once before.

Oh, my FA report stated (besides NO BOOT) that some of the capacitors were swollen....

Cisco???