MD110 almost total failure

[mrlugz]

Over the weekend we lost connectivity between most of our 9 LIMs.
They are connected via E1 links, all our own, running over an OTN service.

printout below.

Im a bit green when it comes to hardware side of it, if someone could give me some pointers I would really appreciate it.


allip;
ALARM LIST


IDENTITY:
VERSION: CXP1010102/4/TSWSP5GB1.46/R6A


DATE: 03OCT15 TIME: 08:52:40


CODE CLASS NOAF EXPLANATION


26 2 1 PERIODICAL SYSTEM DUMP HAS FAILED
30 1 1 INCREMENTATION ALARM FOR ALARM CLASS 0
32 3 1 INCREMENTATION ALARM FOR ALARM CLASS 2
33 4 1 INCREMENTATION ALARM FOR ALARM CLASS 3
34 3 4 LIM OUT OF ORDER
35 4 5 TRAFFIC HANDLING COMMON FUNCTION OUT OF ORDER
37 2 10 ORDINARY COMMON FUNCTION OUT OF ORDER
42 3 7 RELOAD OF EXCHANGE DATA IN LIM HAS FAILED
45 1 1 LIM RELOADED AND RESTARTED
50 1 4 RESTART OF LIM HAS FAILED
260 1 6 FAULT ON LINE TO INTERWORKING PABX OR PUBLIC EXCHANGE
274 1 1 DIGITAL TRUNK, CLOCK MALFUNCTION (SLIP)
278 3 2 CIL DATA LOST FOR ONE OR MORE OUTPUT UNITS


ALREI NOT DONE!


END
/* SUCCEED 08:44:50 */
/* MML_TTY_COMMAND 08:49:16
allop;
ALARM LOG


IDENTITY:
VERSION: CXP1010102/4/TSWSP5GB1.46/R6A


DATE: 03OCT15 TIME: 08:57:08


CLASS: 4


35 TRAFFIC HANDLING COMMON FUNCTION OUT OF ORDER
DATE TIME ALP NOIF UNIT
******* 00:02:02 13 337 DDEM


35 TRAFFIC HANDLING COMMON FUNCTION OUT OF ORDER
DATE TIME ALP NOIF UNIT
******* 00:02:02 14 337 DDGM


35 TRAFFIC HANDLING COMMON FUNCTION OUT OF ORDER
DATE TIME ALP NOIF UNIT
******* 00:02:02 15 337 DMGM


35 TRAFFIC HANDLING COMMON FUNCTION OUT OF ORDER
DATE TIME ALP NOIF UNIT
******* 00:02:04 17 337 DPM


35 TRAFFIC HANDLING COMMON FUNCTION OUT OF ORDER
DATE TIME ALP NOIF UNIT
******* 00:02:04 26 337 DTM


33 INCREMENTATION ALARM FOR ALARM CLASS 3
DATE TIME ALP NOIF
03OCT15 08:49:26 67 8




CLASS: 3


32 INCREMENTATION ALARM FOR ALARM CLASS 2
DATE TIME ALP NOIF
******* 00:02:05 27 7


34 LIM OUT OF ORDER
DATE TIME ALP NOIF EQU INF1
01OCT15 00:20:59 28 337 002-*-**-** 0


34 LIM OUT OF ORDER
DATE TIME ALP NOIF EQU
01OCT15 00:21:02 29 337 003-*-**-**


34 LIM OUT OF ORDER
DATE TIME ALP NOIF EQU
01OCT15 00:21:05 30 337 005-*-**-**


278 CIL DATA LOST FOR ONE OR MORE OUTPUT UNITS
DATE TIME ALP NOIF UNIT EQU INF1
01OCT15 00:36:12 47 234 CLOP 004-*-**-** 2


278 CIL DATA LOST FOR ONE OR MORE OUTPUT UNITS
DATE TIME ALP NOIF UNIT EQU INF1
03OCT15 07:41:29 68 6 CLOP 001-*-**-** 2


34 LIM OUT OF ORDER
DATE TIME ALP NOIF EQU
03OCT15 08:49:00 59 140 004-*-**-**


42 RELOAD OF EXCHANGE DATA IN LIM HAS FAILED
DATE TIME ALP NOIF EQU INF1 INF2 INF3
03OCT15 08:49:25 46 663 002-*-**-** 97 1 96


42 RELOAD OF EXCHANGE DATA IN LIM HAS FAILED
DATE TIME ALP NOIF EQU INF1 INF2 INF3
03OCT15 08:49:25 48 663 003-*-**-** 97 1 96


42 RELOAD OF EXCHANGE DATA IN LIM HAS FAILED
DATE TIME ALP NOIF EQU INF1 INF2 INF3
03OCT15 08:49:25 49 663 004-*-**-** 97 1 96


42 RELOAD OF EXCHANGE DATA IN LIM HAS FAILED
DATE TIME ALP NOIF EQU INF1 INF2 INF3
03OCT15 08:49:25 50 663 005-*-**-** 97 1 96


42 RELOAD OF EXCHANGE DATA IN LIM HAS FAILED
DATE TIME ALP NOIF EQU INF1 INF2 INF3
03OCT15 08:49:26 51 650 006-*-**-** 97 1 96


42 RELOAD OF EXCHANGE DATA IN LIM HAS FAILED
DATE TIME ALP NOIF EQU INF1 INF2 INF3
03OCT15 08:49:26 52 648 007-*-**-** 97 1 96


42 RELOAD OF EXCHANGE DATA IN LIM HAS FAILED
DATE TIME ALP NOIF EQU INF1 INF2 INF3
03OCT15 08:49:26 53 636 008-*-**-** 97 1 96




CLASS: 2


37 ORDINARY COMMON FUNCTION OUT OF ORDER
DATE TIME ALP NOIF UNIT
******* 00:02:02 12 337 FAM2


37 ORDINARY COMMON FUNCTION OUT OF ORDER
DATE TIME ALP NOIF UNIT
******* 00:02:02 16 337 CTLMP


37 ORDINARY COMMON FUNCTION OUT OF ORDER
DATE TIME ALP NOIF UNIT
******* 00:02:04 18 337 CLOM


37 ORDINARY COMMON FUNCTION OUT OF ORDER
DATE TIME ALP NOIF UNIT
******* 00:02:04 19 337 DIM


37 ORDINARY COMMON FUNCTION OUT OF ORDER
DATE TIME ALP NOIF UNIT
******* 00:02:04 20 337 RA1M


37 ORDINARY COMMON FUNCTION OUT OF ORDER
DATE TIME ALP NOIF UNIT
******* 00:02:04 21 337 AOM1


37 ORDINARY COMMON FUNCTION OUT OF ORDER
DATE TIME ALP NOIF UNIT
******* 00:02:04 22 337 AOM2


37 ORDINARY COMMON FUNCTION OUT OF ORDER
DATE TIME ALP NOIF UNIT
******* 00:02:04 23 337 AUM1


37 ORDINARY COMMON FUNCTION OUT OF ORDER
DATE TIME ALP NOIF UNIT
******* 00:02:04 24 337 AUM2


37 ORDINARY COMMON FUNCTION OUT OF ORDER
DATE TIME ALP NOIF UNIT
******* 00:02:04 25 337 ROM


26 PERIODICAL SYSTEM DUMP HAS FAILED
DATE TIME ALP NOIF INF1
03OCT15 02:16:50 56 324 97




CLASS: 1


260 FAULT ON LINE TO INTERWORKING PABX OR PUBLIC EXCHANGE
DATE TIME ALP NOIF UNIT EQU BRDID INF1 INF2
******* 00:00:57 4 2 TLP1 001-2-70-01 75 96 1


260 FAULT ON LINE TO INTERWORKING PABX OR PUBLIC EXCHANGE
DATE TIME ALP NOIF UNIT EQU BRDID INF1 INF2
******* 00:00:57 5 2 TLP1 001-2-70-02 75 94 1


260 FAULT ON LINE TO INTERWORKING PABX OR PUBLIC EXCHANGE
DATE TIME ALP NOIF UNIT EQU BRDID INF1 INF2
******* 00:00:57 6 1 TLP1 001-2-70-03 75 92 1


260 FAULT ON LINE TO INTERWORKING PABX OR PUBLIC EXCHANGE
DATE TIME ALP NOIF UNIT EQU BRDID INF1 INF2
******* 00:01:02 7 2 TLP1 001-2-72-01 75 95 1


260 FAULT ON LINE TO INTERWORKING PABX OR PUBLIC EXCHANGE
DATE TIME ALP NOIF UNIT EQU BRDID INF1 INF2
******* 00:01:02 8 2 TLP1 001-2-72-02 75 93 1


260 FAULT ON LINE TO INTERWORKING PABX OR PUBLIC EXCHANGE
DATE TIME ALP NOIF UNIT EQU BRDID INF1 INF2
******* 00:01:02 9 1 TLP1 001-2-72-03 75 91 1


45 LIM RELOADED AND RESTARTED
DATE TIME ALP NOIF EQU INF1
******* 00:01:17 10 1 001-*-**-** 0


30 INCREMENTATION ALARM FOR ALARM CLASS 0
DATE TIME ALP NOIF
02OCT15 09:49:44 65 999


274 DIGITAL TRUNK, CLOCK MALFUNCTION (SLIP)
DATE TIME ALP NOIF UNIT EQU BRDID INF1
03OCT15 08:14:01 55 24 TLP50 004-2-00-00 48 263


50 RESTART OF LIM HAS FAILED
DATE TIME ALP NOIF EQU INF1
03OCT15 08:48:46 33 336 004-*-**-** 1


50 RESTART OF LIM HAS FAILED
DATE TIME ALP NOIF EQU INF1
03OCT15 08:48:46 34 336 006-*-**-** 1


50 RESTART OF LIM HAS FAILED
DATE TIME ALP NOIF EQU INF1
03OCT15 08:48:47 35 336 007-*-**-** 1


50 RESTART OF LIM HAS FAILED
DATE TIME ALP NOIF EQU INF1
03OCT15 08:48:47 36 336 008-*-**-** 1




CLASS: 0


106 FAULTY I/O BOARD OR CONNECTED I/O DEVICE
DATE TIME ALP NOIF EQU BRDID INF1
******* 00:00:20 2 1 001-0-60-00 106 5
RDATE RTIME
******* 00:02:05


303 DATA LINK ALARM
DATE TIME ALP NOIF UNIT EQU BRDID INF1
******* 00:00:52 3 1 SLP60 001-2-00-00 57 256
RDATE RTIME
******* 00:00:52


88 GROUP SWITCH MODULE CLOCK ALARM
DATE TIME ALP NOIF GSSIDE GSM
******* 00:01:53 11 1 0 0
RDATE RTIME
01OCT15 00:22:08


345 INFORMATION COMPUTER, FAULTY COMMUNICATION CHANNEL
DATE TIME ALP NOIF EQU BRDID
01OCT15 00:21:17 1 3 001-0-60-04 106
RDATE RTIME
01OCT15 00:21:22


283 CONGESTION VALUE FOR KR REACHED
DATE TIME ALP NOIF UNIT EQU INF1 INF2 INF3
01OCT15 00:30:00 32 3 TRP 001-*-**-** 62 597 40
RDATE RTIME
01OCT15 00:45:00


42 RELOAD OF EXCHANGE DATA IN LIM HAS FAILED
DATE TIME ALP NOIF EQU INF1 INF2 INF3
01OCT15 00:30:45 39 2 002-*-**-** 97 1 99
RDATE RTIME
01OCT15 00:40:15


42 RELOAD OF EXCHANGE DATA IN LIM HAS FAILED
DATE TIME ALP NOIF EQU INF1 INF2 INF3
01OCT15 00:30:45 40 2 003-*-**-** 97 1 99
RDATE RTIME
01OCT15 00:40:15


42 RELOAD OF EXCHANGE DATA IN LIM HAS FAILED
DATE TIME ALP NOIF EQU INF1 INF2 INF3
01OCT15 00:30:45 41 2 004-*-**-** 97 1 99
RDATE RTIME
01OCT15 00:40:15


42 RELOAD OF EXCHANGE DATA IN LIM HAS FAILED
DATE TIME ALP NOIF EQU INF1 INF2 INF3
01OCT15 00:30:45 42 2 005-*-**-** 97 1 99
RDATE RTIME
01OCT15 00:40:15


42 RELOAD OF EXCHANGE DATA IN LIM HAS FAILED
DATE TIME ALP NOIF EQU INF1 INF2 INF3
01OCT15 00:30:46 43 2 006-*-**-** 97 1 99
RDATE RTIME
01OCT15 00:40:15


42 RELOAD OF EXCHANGE DATA IN LIM HAS FAILED
DATE TIME ALP NOIF EQU INF1 INF2 INF3
01OCT15 00:30:46 44 2 007-*-**-** 97 1 99
RDATE RTIME
01OCT15 00:40:15


42 RELOAD OF EXCHANGE DATA IN LIM HAS FAILED
DATE TIME ALP NOIF EQU INF1 INF2 INF3
01OCT15 00:30:46 45 2 008-*-**-** 97 1 99
RDATE RTIME
01OCT15 00:40:15


260 FAULT ON LINE TO INTERWORKING PABX OR PUBLIC EXCHANGE
DATE TIME ALP NOIF UNIT EQU BRDID INF1 INF2
02OCT15 08:07:24 57 4 TLP1 001-2-70-00 75 98 1
RDATE RTIME
02OCT15 08:07:24


287 CONGESTION SUPERVISION VALUE REACHED FOR PCM-LINE
DATE TIME ALP NOIF UNIT EQU INF1 INF2 INF3
02OCT15 09:30:00 61 2 GSP 008-*-**-** 1000 599 10
RDATE RTIME
02OCT15 09:45:00


287 CONGESTION SUPERVISION VALUE REACHED FOR PCM-LINE
DATE TIME ALP NOIF UNIT EQU INF1 INF2 INF3
02OCT15 09:30:00 62 3 GSP 007-*-**-** 1000 599 10
RDATE RTIME
02OCT15 09:45:00


287 CONGESTION SUPERVISION VALUE REACHED FOR PCM-LINE
DATE TIME ALP NOIF UNIT EQU INF1 INF2 INF3
02OCT15 09:30:00 63 2 GSP 006-*-**-** 1000 599 10
RDATE RTIME
02OCT15 09:45:00


260 FAULT ON LINE TO INTERWORKING PABX OR PUBLIC EXCHANGE
DATE TIME ALP NOIF UNIT EQU BRDID INF1 INF2
02OCT15 19:53:07 66 1 TLP1 004-2-62-00 75 96 1
RDATE RTIME
02OCT15 19:53:09


34 LIM OUT OF ORDER
DATE TIME ALP NOIF EQU
03OCT15 06:07:13 37 195 007-*-**-**
RDATE RTIME
03OCT15 06:16:53


34 LIM OUT OF ORDER
DATE TIME ALP NOIF EQU
03OCT15 08:38:56 58 106 006-*-**-**
RDATE RTIME
03OCT15 08:48:43


34 LIM OUT OF ORDER
DATE TIME ALP NOIF EQU
03OCT15 08:38:59 38 213 008-*-**-**
RDATE RTIME
03OCT15 08:48:45


34 LIM OUT OF ORDER
DATE TIME ALP NOIF EQU INF2
03OCT15 08:48:51 54 109 007-*-**-** 0
RDATE RTIME
03OCT15 08:48:51


34 LIM OUT OF ORDER
DATE TIME ALP NOIF EQU INF2
03OCT15 08:48:51 60 82 008-*-**-** 0
RDATE RTIME
03OCT15 08:48:51


34 LIM OUT OF ORDER
DATE TIME ALP NOIF EQU INF2
03OCT15 08:48:51 64 37 006-*-**-** 0
RDATE RTIME
03OCT15 08:48:51


24 FAULTY PCM LINK (GJU)
DATE TIME ALP NOIF GSSIDE GSM
03OCT15 08:56:24 31 498 0 0
RDATE RTIME
03OCT15 08:57:06

 

[parnum]

Hi

Lim 1 reloaded and the fault is now cleared Class 0

If the other lims are not back (they show Class 3 Out Of Order), you may need to physically pull the LPU card from each lim and reseat it to get each lim to reload from the LPU in lim 1 again, as a simple restart will try and reload from the buffer unit BUP and fail (as it may have done).

If they're back, do nothing and consider applying the latest Service Pack,

best parnum

 

[mrlugz]

Cheers Parnum,

Turns out we had 3 faulty LPU cards.

Ludicrous as it sounds, they appeared to fail simultaneously.

Ended up getting someone in who had access to some test gear that I did not. Had I not seen the issues personally I would have never believed it possible.

Kind regards

Lugz