CS1K_E SRPT - Continous errors

[mrclean0613]

Greetings ~

I have a CS1K-E kicking out SRPT errors - it looks as if CPU-0 is defective or something on CPU-0 not feeling well. I have attached the HST file to show exactly what's going on.

Looking for some guidance on a resolution >
Would an INI possibly clear this? What happens if there’s more of an issue and the INI seizes the system

Would a cold restart possibly clear this issue? I have no spares – so we’re a bit reluctant to banging on the Nortel if not necessary

Here is a snap shot of the HST file

REQ PRT
TYPE AHST
% SRPT131 CM: Primary client can't connect to the other side.
%
% SRPT027 HB: Cannot detect heartbeat from other core
%
% SRPT025 CPM received stop update message from LCS.
%
% SRPT117 CM: Server connection established.
%
% SRPT110 CM: Primary client connection established.
%
% SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.
%
% SRPT021 LCS: hsp state changed from LcsHspStUp to LcsHspStDown.
%
% SRPT091 HB: Remote side IPL health change:0.
%
% SRPT028 HB: Heartbeat detected from remote core.
%
% SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.
%
% SRPT024 CPM: CPM started protected memory sync.
%
% SRPT023 CPM: CPM completed protected memory sync.
%
% SRPT033 DR: Master is starting disk sync.
%
% SRPT095 HIRM: fmd 0 16 1 on the remote side 1 is Out Of Service.
%
% SRPT026 HB: Remote side health change: 12 .
%
% SRPT043 LCS: Protected memory and disk synchronization complete.
%
% SRPT050 DR: Disk sync completed.
%
% SRPT091 HB: Remote side IPL health change:14.
%
% SRPT096 HIRM: fmd 0 16 1 on the remote side 1 is In Service.
%
% SRPT026 HB: Remote side health change: 20 .
%
% TTY #15 LD 032 ADMIN1 10:24:14 26/7/2011
%
% TTY #15 LD 135 ADMIN1 10:24:18 26/7/2011
%
% TTY #15 LD 020 ADMIN1 10:24:52 26/7/2011
%
% TTY #15 LD 032 ADMIN1 10:25:08 26/7/2011
%
% TIM000 10:25 26/7/2011 CPU 1
%
% TTY #15 LD 011 ADMIN1 10:25:16 26/7/2011
%
% TTY #15 LD 020 ADMIN1 10:27:10 26/7/2011
%
% SRPT118 CM: Server connection lost.
%
% SRPT021 LCS: hsp state changed from LcsHspStUp to LcsHspStDown.
%
% SRPT025 CPM received stop update message from LCS.
%
% SRPT049 DR: Master asked to stop updates.
%
% SRPT131 CM: Primary client can't connect to the other side.
%
% SRPT027 HB: Cannot detect heartbeat from other core
%
% SRPT025 CPM received stop update message from LCS.
%
% SRPT110 CM: Primary client connection established.
%
% SRPT117 CM: Server connection established.
%
% SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.
%
% SRPT021 LCS: hsp state changed from LcsHspStUp to LcsHspStDown.
%
% SRPT091 HB: Remote side IPL health change:0.
%
% SRPT028 HB: Heartbeat detected from remote core.
%
% SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.
%
% SRPT024 CPM: CPM started protected memory sync.
%
% AUD000
%
% SRPT023 CPM: CPM completed protected memory sync.
%
% SRPT033 DR: Master is starting disk sync.
%
% SRPT095 HIRM: fmd 0 16 1 on the remote side 1 is Out Of Service.
%
% SRPT026 HB: Remote side health change: 12 .
%
% SRPT043 LCS: Protected memory and disk synchronization complete.
%
% SRPT050 DR: Disk sync completed.
%
% SRPT091 HB: Remote side IPL health change:14.
%
% SRPT096 HIRM: fmd 0 16 1 on the remote side 1 is In Service.
%
% SRPT026 HB: Remote side health change: 20 .
%
% TTY #15 LD 135 ADMIN1 10:43:52 26/7/2011
%
% TTY #15 LD 020 ADMIN1 10:44:02 26/7/2011
%
% TTY #15 LD 032 ADMIN1 10:44:44 26/7/2011
%
% TTY #15 LD 020 ADMIN1 10:45:50 26/7/2011
%
% TTY #15 LOGGED OUT ADMIN1 10:54 26/7/2011
% SESSION DURATION: 00:51
%
% AUD000
%
% SRPT118 CM: Server connection lost.
%
% SRPT021 LCS: hsp state changed from LcsHspStUp to LcsHspStDown.
%
% SRPT025 CPM received stop update message from LCS.
%
% SRPT049 DR: Master asked to stop updates.
%
% SRPT131 CM: Primary client can't connect to the other side.
%
% SRPT027 HB: Cannot detect heartbeat from other core
%
% SRPT025 CPM received stop update message from LCS.
%
% TTY #15 LOGGED IN ADMIN1 10:58 26/7/2011
%
% TTY #15 LD 020 ADMIN1 10:58:32 26/7/2011
%
% TTY #15 LD 022 ADMIN1 10:58:56 26/7/2011
%
% SRPT117 CM: Server connection established.
%
% SRPT110 CM: Primary client connection established.
%
% SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.
%
% SRPT021 LCS: hsp state changed from LcsHspStUp to LcsHspStDown.
%
% SRPT091 HB: Remote side IPL health change:0.
%
% SRPT028 HB: Heartbeat detected from remote core.
%
% SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.
%
% SRPT024 CPM: CPM started protected memory sync.
%
% TIM000 11:00 26/7/2011 CPU 1
%
% SRPT023 CPM: CPM completed protected memory sync.
%
% SRPT033 DR: Master is starting disk sync.
%
% SRPT095 HIRM: fmd 0 16 1 on the remote side 1 is Out Of Service.
%
% SRPT026 HB: Remote side health change: 12 .
%
% SRPT043 LCS: Protected memory and disk synchronization complete.
%
% SRPT050 DR: Disk sync completed.
%
% SRPT026 HB: Remote side health change: 20 .
%
% SRPT091 HB: Remote side IPL health change:14.
%
% SRPT096 HIRM: fmd 0 16 1 on the remote side 1 is In Service.
%
% TTY #15 LD 022 ADMIN1 11:01:58 26/7/2011
%
% TTY #15 LOGGED OUT ADMIN1 11:05 26/7/2011
% SESSION DURATION: 00:07
%
% ERR057 4 1 9 3 00004060
%
% AUD000
%
% TIM000 11:15 26/7/2011 CPU 1
%
% SRPT118 CM: Server connection lost.
%
% SRPT021 LCS: hsp state changed from LcsHspStUp to LcsHspStDown.
%
% SRPT025 CPM received stop update message from LCS.
%
% SRPT049 DR: Master asked to stop updates.
%
% SRPT131 CM: Primary client can't connect to the other side.
%
% SRPT027 HB: Cannot detect heartbeat from other core
%
% SRPT025 CPM received stop update message from LCS.
%
% SRPT117 CM: Server connection established.
%
% SRPT110 CM: Primary client connection established.
%
% SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.
%
% SRPT021 LCS: hsp state changed from LcsHspStUp to LcsHspStDown.
%
% SRPT091 HB: Remote side IPL health change:0.
%
% SRPT028 HB: Heartbeat detected from remote core.
%
% SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.
%
% SRPT024 CPM: CPM started protected memory sync.
%
% SRPT023 CPM: CPM completed protected memory sync.
%
% SRPT033 DR: Master is starting disk sync.
%
% SRPT095 HIRM: fmd 0 16 1 on the remote side 1 is Out Of Service.
%
% SRPT026 HB: Remote side health change: 12 .
%
% SRPT043 LCS: Protected memory and disk synchronization complete.
%
% SRPT050 DR: Disk sync completed.
%
% SRPT091 HB: Remote side IPL health change:8.
%
% SRPT096 HIRM: fmd 0 16 1 on the remote side 1 is In Service.
%
% SRPT026 HB: Remote side health change: 20 .
%
% SRPT091 HB: Remote side IPL health change:14.
%
% AUD000
%
% SRPT118 CM: Server connection lost.
%
% SRPT021 LCS: hsp state changed from LcsHspStUp to LcsHspStDown.
%
% SRPT025 CPM received stop update message from LCS.
%
% SRPT049 DR: Master asked to stop updates.
%
% SRPT131 CM: Primary client can't connect to the other side.
%
% SRPT027 HB: Cannot detect heartbeat from other core
%
% SRPT025 CPM received stop update message from LCS.
%
% SRPT117 CM: Server connection established.
%
% SRPT110 CM: Primary client connection established.
%
% SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.
%
% SRPT021 LCS: hsp state changed from LcsHspStUp to LcsHspStDown.
%
% SRPT091 HB: Remote side IPL health change:0.
%
% SRPT028 HB: Heartbeat detected from remote core.
%
% SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.
%
% SRPT024 CPM: CPM started protected memory sync.
%
% SRPT023 CPM: CPM completed protected memory sync.
%
% SRPT033 DR: Master is starting disk sync.
%
% SRPT095 HIRM: fmd 0 16 1 on the remote side 1 is Out Of Service.
%
% SRPT026 HB: Remote side health change: 12 .
%
% SRPT043 LCS: Protected memory and disk synchronization complete.
%
% SRPT050 DR: Disk sync completed.
%
% SRPT091 HB: Remote side IPL health change:10.
%
% SRPT096 HIRM: fmd 0 16 1 on the remote side 1 is In Service.
%
% SRPT026 HB: Remote side health change: 20 .
%
% SRPT091 HB: Remote side IPL health change:14.
%
% SRPT118 CM: Server connection lost.
%
% SRPT021 LCS: hsp state changed from LcsHspStUp to LcsHspStDown.
%
% SRPT025 CPM received stop update message from LCS.
%
% SRPT049 DR: Master asked to stop updates.
%
% SRPT131 CM: Primary client can't connect to the other side.
%
% SRPT027 HB: Cannot detect heartbeat from other core
%
% SRPT025 CPM received stop update message from LCS.
%
% TIM000 12:00 26/7/2011 CPU 1
%
% SRPT110 CM: Primary client connection established.
%
% SRPT117 CM: Server connection established.
%
% SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.
%
% SRPT021 LCS: hsp state changed from LcsHspStUp to LcsHspStDown.
%
% SRPT091 HB: Remote side IPL health change:0.
%
% SRPT028 HB: Heartbeat detected from remote core.
%
% SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.
%
% SRPT024 CPM: CPM started protected memory sync.
%
% AUD000
%
% SRPT023 CPM: CPM completed protected memory sync.
%
% SRPT033 DR: Master is starting disk sync.
%
% SRPT095 HIRM: fmd 0 16 1 on the remote side 1 is Out Of Service.
%
% SRPT026 HB: Remote side health change: 12 .
%
% SRPT091 HB: Remote side IPL health change:14.
%
% SRPT043 LCS: Protected memory and disk synchronization complete.
%
% SRPT050 DR: Disk sync completed.
%
% SRPT096 HIRM: fmd 0 16 1 on the remote side 1 is In Service.
%
% SRPT026 HB: Remote side health change: 20 .
%
% SRPT118 CM: Server connection lost.
%
% SRPT021 LCS: hsp state changed from LcsHspStUp to LcsHspStDown.
%
% SRPT025 CPM received stop update message from LCS.
%
% SRPT049 DR: Master asked to stop updates.
%
% SRPT131 CM: Primary client can't connect to the other side.
%
% SRPT027 HB: Cannot detect heartbeat from other core
%
% SRPT025 CPM received stop update message from LCS.
%
% SRPT117 CM: Server connection established.
%
% SRPT110 CM: Primary client connection established.
%
% SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.
%
% SRPT021 LCS: hsp state changed from LcsHspStUp to LcsHspStDown.
%
% SRPT091 HB: Remote side IPL health change:0.
%
% SRPT028 HB: Heartbeat detected from remote core.
%
% SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.
%
% SRPT024 CPM: CPM started protected memory sync.
%
% SRPT023 CPM: CPM completed protected memory sync.
%
% SRPT033 DR: Master is starting disk sync.
%
% SRPT095 HIRM: fmd 0 16 1 on the remote side 1 is Out Of Service.
%
% SRPT026 HB: Remote side health change: 12 .
%
% SRPT043 LCS: Protected memory and disk synchronization complete.
%
% SRPT050 DR: Disk sync completed.
%
% SRPT091 HB: Remote side IPL health change:12.
%
% SRPT096 HIRM: fmd 0 16 1 on the remote side 1 is In Service.
%
% SRPT026 HB: Remote side health change: 20 .
%
% SRPT091 HB: Remote side IPL health change:14.
%
% AUD000
%
% TTY #15 LOGGED IN ADMIN1 12:35 26/7/2011
%
% TTY #15 LD 020 ADMIN1 12:35:17 26/7/2011
%
% TTY #15 LD 022 ADMIN1 12:35:25 26/7/2011
%
% SRPT118 CM: Server connection lost.
%
% SRPT021 LCS: hsp state changed from LcsHspStUp to LcsHspStDown.
%
% SRPT025 CPM received stop update message from LCS.
%
% SRPT049 DR: Master asked to stop updates.
%
% SRPT131 CM: Primary client can't connect to the other side.
%
% SRPT027 HB: Cannot detect heartbeat from other core
%
% SRPT025 CPM received stop update message from LCS.
%
% TTY #15 LOGGED OUT ADMIN1 12:44 26/7/2011
% SESSION DURATION: 00:09
%
% AUD000
%
% SRPT117 CM: Server connection established.
%
% SRPT110 CM: Primary client connection established.
%
% SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.
%
% SRPT021 LCS: hsp state changed from LcsHspStUp to LcsHspStDown.
%
% SRPT091 HB: Remote side IPL health change:0.
%
% SRPT028 HB: Heartbeat detected from remote core.
%
% SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.
%
% SRPT024 CPM: CPM started protected memory sync.
%
% SRPT023 CPM: CPM completed protected memory sync.
%
% SRPT033 DR: Master is starting disk sync.
%
% SRPT095 HIRM: fmd 0 16 1 on the remote side 1 is Out Of Service.
%
% SRPT026 HB: Remote side health change: 12 .
%
% SRPT091 HB: Remote side IPL health change:14.
%
% SRPT043 LCS: Protected memory and disk synchronization complete.
%
% SRPT050 DR: Disk sync completed.
%
% SRPT096 HIRM: fmd 0 16 1 on the remote side 1 is In Service.
%
% SRPT026 HB: Remote side health change: 20 .
%
% TIM000 13:00 26/7/2011 CPU 1
%
% TTY #05 LOGGED IN ADMIN1 13:00 26/7/2011
%
% TTY #05 LD 135 ADMIN1 13:00:15 26/7/2011
%
% TTY #05 LD 043 ADMIN1 13:08:47 26/7/2011
%
% TEMU207 Backup process to local Removable Media Device ended successfully.
%
% EDD000
%
%
% SRPT117 CM: Server connection established.
%
% SRPT110 CM: Primary client connection established.
%
% SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.
%
% SRPT021 LCS: hsp state changed from LcsHspStUp to LcsHspStDown.
%
% SRPT091 HB: Remote side IPL health change:0.
%
% SRPT028 HB: Heartbeat detected from remote core.
%
% SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.
%
% SRPT024 CPM: CPM started protected memory sync.
%
% EDD000
%
%
% EDD000
%
%
% SRPT118 CM: Server connection lost.
%
% SRPT131 CM: Primary client can't connect to the other side.
%
% SRPT027 HB: Cannot detect heartbeat from other core
%
% TEMU207 Backup process to local Removable Media Device ended successfully.
%
% EDD000
%
%
% SRPT023 CPM: CPM completed protected memory sync.
%
% SRPT025 CPM received stop update message from LCS.
%
% TTY #05 LOGGED OUT ADMIN1 13:13 26/7/2011
% SESSION DURATION: 00:13
%
% AUD000
%
% TIM000 13:16 26/7/2011 CPU 1
%
% ERR057 4 1 9 3 00004060
%
% SRPT110 CM: Primary client connection established.
%
% SRPT117 CM: Server connection established.
%
% SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.
%
% SRPT021 LCS: hsp state changed from LcsHspStUp to LcsHspStDown.
%
% SRPT028 HB: Heartbeat detected from remote core.
%
% SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.
%
% SRPT024 CPM: CPM started protected memory sync.
%
% SRPT023 CPM: CPM completed protected memory sync.
%
% SRPT033 DR: Master is starting disk sync.
%
% SRPT095 HIRM: fmd 0 16 1 on the remote side 1 is Out Of Service.
%
% SRPT026 HB: Remote side health change: 12 .
%
% SRPT091 HB: Remote side IPL health change:14.
%
% SRPT043 LCS: Protected memory and disk synchronization complete.
%
% SRPT050 DR: Disk sync completed.
%
% SRPT096 HIRM: fmd 0 16 1 on the remote side 1 is In Service.
%
% SRPT026 HB: Remote side health change: 20 .
%
% SRPT118 CM: Server connection lost.
%
% SRPT021 LCS: hsp state changed from LcsHspStUp to LcsHspStDown.
%
% SRPT025 CPM received stop update message from LCS.
%
% SRPT049 DR: Master asked to stop updates.
%
% SRPT131 CM: Primary client can't connect to the other side.
%
% SRPT027 HB: Cannot detect heartbeat from other core
%
% SRPT025 CPM received stop update message from LCS.
%
% SRPT117 CM: Server connection established.
%
% SRPT110 CM: Primary client connection established.
%
% SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.
%
% SRPT021 LCS: hsp state changed from LcsHspStUp to LcsHspStDown.
%
% SRPT091 HB: Remote side IPL health change:0.
%
% SRPT028 HB: Heartbeat detected from remote core.
%
% SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.
%
% SRPT024 CPM: CPM started protected memory sync.
%
% SRPT023 CPM: CPM completed protected memory sync.
%
% SRPT033 DR: Master is starting disk sync.
%
% SRPT095 HIRM: fmd 0 16 1 on the remote side 1 is Out Of Service.
%
% SRPT026 HB: Remote side health change: 12 .
%
% SRPT043 LCS: Protected memory and disk synchronization complete.
%
% SRPT050 DR: Disk sync completed.
%
% SRPT026 HB: Remote side health change: 20 .
%
% SRPT091 HB: Remote side IPL health change:14.
%
% SRPT096 HIRM: fmd 0 16 1 on the remote side 1 is In Service.
%
% SRPT118 CM: Server connection lost.
%
% SRPT021 LCS: hsp state changed from LcsHspStUp to LcsHspStDown.
%
% SRPT025 CPM received stop update message from LCS.
%
% SRPT049 DR: Master asked to stop updates.
%
% SRPT131 CM: Primary client can't connect to the other side.
%
% SRPT027 HB: Cannot detect heartbeat from other core
%
% SRPT025 CPM received stop update message from LCS.
%
% AUD000
%
% TTY #15 LOGGED IN ADMIN1 13:46 26/7/2011
%
% TTY #15 LD 020 ADMIN1 13:46:14 26/7/2011
%
% TTY #15 LD 022 ADMIN1 13:46:24 26/7/2011


Any sugestions are truly appreciated!

thank you,

mrclean613

 

[djk205]

Have you tried a different elan port?

 

[mrclean0613]

Hey djk205 ~

Are you saying these errors are pointing to an elan port? the errors appear to be pointing to the CPU-0

 

[steeviedude]

Is the HSP direct connected to cores or routed over network?
Do you have access to PDT for the RD Logs?

 

[mrclean0613]

Hmmm ~

Not sure of these acronyms...

what is HSP?

You have already probably figured out my answer to the next question. I do not have access to PDT

 

[steeviedude]

HSP is the cable providing direct connection between the two processors.

LD 137 STAT HSP & STAT ELNK see if shows any errors/collisions/Broadcast storms etc on link and check ELAN ports on network.

 

[bignose21]

Hi,
Looking at the errors the offline side is coming up, the realises it is the standby does an INI comes up properly, does a memory Sync and Disk Sync, then comes fully on-line:

% SRPT026 HB: Remote side health change: 20

At that point the process starts again.

Just as a matter of interest un-plug the ELAN cable from the Standby (The one in constant reboot), does it stabilize?

If not I would leave the ELAN out until it is stable that way it will never win on health state to take over the switch.

If it didn't work and it is still restarting try a Sysload (Reset Button) on the standby. If still no Joy you might be looking at a reload of the CPU software or a Hardware issue.

What level of software is the switch on and what CPU type?

 

[bhassell2000]

I was thinking software corruption and reload, possibly changing cpu card

Thanks,

Buddy

Linked in Profile

http://www.linkedin.com/in/buddyhassell

 

[mrclean0613]

Ok...

Here's what I have for LD 137 STAT HSP & STAT ELNK


.stat HSP
LCS HSP STATE is DOWN
HSP LINK CARRIER: OK
Auto Negotiation: Enabled
Auto Negotiation Completed: YES
Actual Line Speed: 1000 Mbps
Actual Duplex Mode: Full Duplex

Ethernet (gei unit number 1):
Internet address: 127.2.0.2
Broadcast address: 127.255.255.255
Ethernet address: 00:c0:8b:0a:2e:83
Netmask: 0xff000000; Subnetmask: 0xff000000
63826009 packets received; 69584678 packets sent
0 input errors; 0 output errors
0 collisions
.
AUD000


CIOD000
.STAT ELNK

ELNK ENABLED
Auto Negotiation: Enabled
Auto Negotiation Completed: YES
Actual Line Speed: 100 Mbps
Actual Duplex Mode: Full Duplex

Ethernet (gei unit number 0):
Host: CALLSERVER_0
Internet address: 192.168.1.100
Broadcast address: 192.168.1.255
Ethernet address: 00:c0:8b:0a:2e:82
Netmask: 0xffffff00 ; Subnetmask: 0xffffff00
1367055572 packets received; -2003154843 packets sent
0 input errors; 0 output errors
0 collisions

 

[steeviedude]

Follow suggestions above from bignose21