Failure to EDD and SCPU

[srcvh1]

I think I have an interesting one here:

I have a customer that has not been backing up for awhile now. After confirming that and receiving multiple errors from the EDD. I went to 135 and tried to swap the Cores. This is what happened:

Starting CCBR backup to "/u/ccbr/ccbr.gz":
CCBR backup failed!

TEMU020 Errno 0x380003: Failure opening file /u/ipmg/mgc

TEMU111 Error occured backing up internal data base


Backing up reten.bkp

Starting database backup
to local Removable Media Device
Database backup failed!

TEMU206 local Removable Media Device is not available

TEMU062 Database backup to removable media failed

EDD007

****
>ld 135
scpu

SRPT0018
LCS: Graceful switchover not executed. Local health is better than
remote health.

Severity: Info


Just wondering if these issues may be related or if I just stumbled onto another issue while troubleshooting the first. Any ideas regarding both or either one of these issues would be greatly appreciated.

Scott

 

[KCFLHRC]

They could be related, stat the health or look at the system utility for health. There was a patch for the TEMU020 depending on what release and machine type you are on. They are probably 2 separate issues. I would also do a stat cpu ion ld 135 and see if they are redundant and this will show the health also.

 

[srcvh1]

KCFLHRC,

I have done a STAT on the health, CPU, and HSP. Here is what it comes back with:

.stat health
Local (Side 1, Active, Redundant):
Components without TIER 1 Health contribution:
==================================================
disp 1 15 1:In Service
sio2 1 15 1:In Service
cp 1 16:In Service
ipb 1:In Service

TIER 1 Health Count Breakdown:
=====================================
sio8 1 16 1: 0002
sio8 1 16 2: 0002
sutl 1 15: 0002
strn 1 15: 0002
xsmp 1 15 1: 0002
cnib 1 9: 0002
cnip 1 9 0: 0002
cnip 1 9 1: 0000 (OutOfService)
fmd 1 16 1: 0008
eth 1 16 0: 0002
Local TIER 1 Health Total: 24


TIER 2 Health Count Breakdown:
=======================================
ELAN 16 IP : 137.135.128.190 Health = 2

Local AML over ELAN Total Health:2
Local Total IPL Health = 2

IPL connection history:1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Local TIER 2 Health Total:4

Remote (Side 0, Inactive, Redundant):
Components without TIER 1 Health contribution:
==================================================
disp 0 15 1:In Service
sio2 0 15 1:In Service
cp 0 16:In Service
ipb 0:In Service

TIER 1 Health Count Breakdown:
=====================================
sio8 0 16 1: 0002
sio8 0 16 2: 0002
sutl 0 15: 0002
strn 0 15: 0002
xsmp 0 15 1: 0002
cnib 0 9: 0002
cnip 0 9 0: 0002
cnip 0 9 1: 0000 (OutOfService)
fmd 0 16 1: 0008
eth 0 16 0: 0002
Remote TIER 1 Health Total: 24


TIER 2 Health Count Breakdown:
=======================================
ELAN 16 IP : 137.135.128.190 Health = 2

Local AML over ELAN Total Health:2
Remote Total IPL health = 0

Remote TIER 2 Health Total:2

.stat health aml
Local (Side 1, Active, Redundant):
ELAN 16 IP : 137.135.128.190 Health = 2

Local AML over ELAN Total Health:2

Remote (Side 0, Inactive, Redundant):
ELAN 16 IP : 137.135.128.190 Health = 2

Local AML over ELAN Total Health:2


.stat cpu

cp 1 16 PASS -- ENBL

SYSTEM STATE = REDUNDANT
DISK STATE = REDUNDANT
HEALTH = 24
VERSION = Jun 2 2009, 17:46:52
Side = 1, DRAM SIZE = 512 MBytes

cp 0 16 PASS -- STDBY

SYSTEM STATE = REDUNDANT
DISK STATE = REDUNDANT
HEALTH = 24
VERSION = Jun 2 2009, 17:46:52
Side = 0, DRAM SIZE = 512 MBytes

=========================================
Summary of Local System Resource (side 1)
=========================================
File Descriptors
-----------------
alloc 66
free 1982
total 2048

Unprotected Heap (bytes)
------------------------
alloc 46394976
free 199371376
total 245766352

Protected Heap (bytes)
----------------------
alloc 1188944
free 3005360
total 4194304
.stat health
Local (Side 1, Active, Redundant):
Components without TIER 1 Health contribution:
==================================================
disp 1 15 1:In Service
sio2 1 15 1:In Service
cp 1 16:In Service
ipb 1:In Service

TIER 1 Health Count Breakdown:
=====================================
sio8 1 16 1: 0002
sio8 1 16 2: 0002
sutl 1 15: 0002
strn 1 15: 0002
xsmp 1 15 1: 0002
cnib 1 9: 0002
cnip 1 9 0: 0002
cnip 1 9 1: 0000 (OutOfService)
fmd 1 16 1: 0008
eth 1 16 0: 0002
Local TIER 1 Health Total: 24


TIER 2 Health Count Breakdown:
=======================================
ELAN 16 IP : 137.135.128.190 Health = 2

Local AML over ELAN Total Health:2
Local Total IPL Health = 2

IPL connection history:1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1

Local TIER 2 Health Total:4

Remote (Side 0, Inactive, Redundant):
Components without TIER 1 Health contribution:
==================================================
disp 0 15 1:In Service
sio2 0 15 1:In Service
cp 0 16:In Service
ipb 0:In Service

TIER 1 Health Count Breakdown:
=====================================
sio8 0 16 1: 0002
sio8 0 16 2: 0002
sutl 0 15: 0002
strn 0 15: 0002
xsmp 0 15 1: 0002
cnib 0 9: 0002
cnip 0 9 0: 0002
cnip 0 9 1: 0000 (OutOfService)
fmd 0 16 1: 0008
eth 0 16 0: 0002
Remote TIER 1 Health Total: 24


TIER 2 Health Count Breakdown:
=======================================
ELAN 16 IP : 137.135.128.190 Health = 2

Local AML over ELAN Total Health:2
Remote Total IPL health = 0

Remote TIER 2 Health Total:2

****
>ld 137
CIOD000
.stat hsp
LCS HSP STATE is UP
HSP LINK CARRIER: OK
Auto Negotiation: Enabled
Auto Negotiation Completed: YES
Actual Line Speed: 1000 Mbps
Actual Duplex Mode: Full Duplex

Ethernet (gei unit number 1):
Internet address: 127.2.0.2
Broadcast address: 127.255.255.255
Ethernet address: 00:c0:8b:0b:ba:03
Netmask: 0xff000000; Subnetmask: 0xff000000
1805697085 packets received; 3408148181 packets sent
0 input errors; 0 output errors
0 collisions

Here is the Rls and machine type:

REQ issp

12/12/13 11:13:05
TID: 430207

VERSION 3521

System type is - Communication Server 1000M SG/CP PIV
CP PIV - Pentium M 1.1 GHz

RELEASE 6
ISSUE 00 R +
IDLE_SET_DISPLAY
DepList 1: core Issue: 02(created: 2010-09-28 13:43:36 (est))

MDP>LAST SUCCESSFUL MDP REFRESH :2010-10-02 07:14:37(Local Time)
MDP>USING DEPLIST ZIP FILE DOWNLOADED :2010-10-01 17:59:00(est)
SYSTEM HAS NO USER SELECTED PEPS IN-SERVICE


LOADWARE VERSION: PSWV 100

INSTALLED LOADWARE PEPS : 0

Everything looks in order to me. I was even able to ping both cores without a problem.

 

[KCFLHRC]

I would tend to agree, everything appears to be in order. However, look at your last MDP refresh, it was over 3 years ago. I would split the cores, update the deplist individually on each core, sysload and then join the cores. My guess is your trouble will go away. Make sure you look at the file creation date on the Compact Flash for the backup and see when it was last modified. That will tell you when the last backup was actually completed. Not the folder date but the date on the actual files within the backup folder.

 

[KCFLHRC]

Do you have access to the MIRRC on the ESPL website. If not do an MDP ISSP in LD 143 and paste your results back here and I will do it for you and see how many patches you are missing.

 

[srcvh1]

I have done that. I already noticed the patch status. That was going to be mY first move, updating the deplist. There are over 150 new patches on 6.0 since the last MDP REFRESH.

I was just trying to see if there might be something deeper. It's this that concerns me:

Starting CCBR backup to "/u/ccbr/ccbr.gz":
CCBR backup failed!

TEMU020 Errno 0x380003: Failure opening file /u/ipmg/mgc

TEMU111 Error occured backing up internal data base

I think this is something that is corrupted. I know there is a way to go into PDT to correct it but I am unsure of the path.

Anyway, I will update the deplist first and cross my fingers that it solves all issues. I'll keep you posted.

Thanks, Scott

 

[KCFLHRC]

Definitely look at those file creation dates on the Compact Flash. If for some reason you have to reload the software that will be the database you have to use.

 

[KCFLHRC]

Good luck, let us know the outcome.