System INI without warning 1

[TelcoChic]

I have a Opt81 DSN system that will INI 3 times per day during normal office hours (does not happen after hours or on weekends). I have captured history data and there is no warning before the system goes down. Can someone please take a look at the INI messages and tell me if they see anything that might help me determine the issue??
Thanks!

DTC001


TIM000 08:00 27/8/2010 CPU 1

INI000 00000000 08 03 24 00000000 00000000 00000000 00000000 1 1 0000007D
DSET000 DOWN LOAD 0 0 0 0 0 0 0 82963

INI002 0000008B 0000008C 00000080

INI003 00000092

INI007 00000076 00000077 0000007A 0000007B 0000007C 0000007D

INI009 00000092

INI010
ACDR ACTIVATED

DCH1014 20 036C048A 00000303

DCH055 DCH: 20 RLS CONFIRM TIME: 8:03:24 8/27/2010

DCH054 DCH: 20 EST CONFIRM TIME: 8:03:24 8/27/2010

CSA003 16 8:03:42 27/08/2010

CSA105 16, 8:03:42 27/08/2010

CSA104 16, 8:04:02 27/08/2010

CSA003 17 8:04:02 27/08/2010

DCH1014 2 036C1701 00000106

DCH1014 5 036C11D7 00000106

DCH1014 6 036C0F5A 00000106

DCH1014 2 036C1701 00000106

DCH1014 5 036C11D7 00000106

DCH1014 6 036C0F5A 00000106

DCH1014 2 036C1701 00000106

DCH1014 5 036C11D7 00000106

DCH1014 6 036C0F5A 00000106

CDN002 16 1600 08 04 02

CDN002 16 1601 08 04 02

CDN002 16 1602 08 04 02

CDN002 16 1603 08 04 02

CDN002 16 1604 08 04 02

DCH054 DCH: 10 EST CONFIRM TIME: 8:04:02 8/27/2010

DCH054 DCH: 11 EST CONFIRM TIME: 8:04:02 8/27/2010

DCH054 DCH: 6 EST CONFIRM TIME: 8:04:02 8/27/2010

DCH054 DCH: 5 EST CONFIRM TIME: 8:04:02 8/27/2010

CSA105 17, 8:04:02 27/08/2010

BUG105
BUG105 : 50 17
BUG105 + 00AFA0A8 00AF469C 00AF384C 00AF37C3 01045B20
BUG105 + 01035FEB 010315BF 0101C617 0101B55B 0101B4CD
BUG105 + 0101B433 0100FA4A 00C70552 00C6EB6A 00C6E998
BUG105 + 00C3F269 00C3EC09 00AC7FD7 00AB8DCD 00AB65E6
BUG105 + 00AB2BDB 00AB1B75 00AADA25 01738BE8 00CA57F2
BUG105 + 09A60243 00CA4231 00CA2313 00C6EA01 0101BBD6
BUG105 + 0101BAEF 0101B697 0101B515 0101B4CD 0101B433
BUG105 + 010144D3 01014370 010119F8 010101BB 0100FA1E
BUG105 + 007FF042 00809401 0080934D 008070F0 007FDA36
BUG105 + 00E36FAF 00E33D60 00E33A66 019779EC 01970C91
BUG105 + 01B53921

DCH054 DCH: 8 EST CONFIRM TIME: 8:04:04 8/27/2010

DCH054 DCH: 2 EST CONFIRM TIME: 8:04:04 8/27/2010

SRPT181 XSM: Alarm signal detected - check the Power/Fan module or the System Monitor mo
dule (XSM).

BUG1374
BUG1374 : 00003284
BUG1374 + 00AFA0A8 01060CE2 01060BBC 01049E84 01922416
BUG1374 + 0191D09F 0191CE99 018B010B 018AFB0D 018E26A5
BUG1374 + 018E21C9 018E2123 018E0137 018E0085 00E3705D
BUG1374 + 00E33D60 00E33A66 019779EC 01970C91 01B53921

BUG1374
BUG1374 : 00003284
BUG1374 + 00AFA0A8 018BF643 018B80FF 018B0263 018AFB0D
BUG1374 + 018E26A5 018E21C9 018E2123 018E0137 018E0085
BUG1374 + 00E3705D 00E33D60 00E33A66 019779EC 01970C91
BUG1374 + 01B53921

CNI201 CNI 1 9 : Watchdog timeout detected count = 2

SRPT110 CM: Primary client connection established.

SRPT117 CM: Server connection established.

SRPT028 HB: Heartbeat detected from remote core.

SRPT026 HB: Remote side health change: 26 .

SRPT026 HB: Local side health change: 26 .

SRPT021 LCS: hsp state changed from LcsHspStDown to LcsHspStUp.

SRPT751 INI 1: starting INI on side 1 due to System Restart
Previous INI: side 1 at 26/ 8/10 15:31:34
INIs since switch-over (26/ 8/10 11:06:21): 5, 3
INIs since cold start (24/ 8/10 10:01:35): 10, 6

SRPT026 HB: Local side health change: 24 .

PRI101 44 23 CAUSE 82

SRPT026 HB: Local side health change: 22 .

SRPT024 CPM: CPM started protected memory sync.

SRPT026 HB: Local side health change: 24 .

SRPT026 HB: Local side health change: 26 .

SRPT091 HB: Local side IPL health change:4.

ELAN014 ELAN 0 host IP=10.1.200.50 is enabled

SRPT752 INI 1: INI completed in 6 seconds

ELAN019 ELAN Server enabled after M1 INIT

ELAN014 ELAN 0 host IP=10.1.200.48 is enabled

SRPT091 HB: Remote side IPL health change:12.

SRPT091 HB: Local side IPL health change:12.

SRPT023 CPM: CPM completed protected memory sync.

SRPT033 DR: Master is starting disk sync.

ELAN014 ELAN 16 host IP=10.1.200.6 is enabled

SRPT095 HIRM: cmdu 0 16 1 on the remote side 0 is Out Of Service.

SRPT093 AML: local side AML connection 16 to 10.1.200.6 health change:2.

SRPT091 HB: Local side AML health change:2.

SRPT026 HB: Remote side health change: 18 .

ELAN014 ELAN 0 host IP=10.1.200.59 is enabled

ELAN014 ELAN 0 host IP=10.1.200.55 is enabled

ELAN014 ELAN 0 host IP=10.1.200.46 is enabled

ELAN014 ELAN 0 host IP=10.1.200.47 is enabled

PRI382 3 78

SRPT092 AML: remote side AML connection 16 to 10.1.200.6 health change: 2.

PRI382 7 78

SRPT091 HB: Remote side AML health change:2.

ELAN014 ELAN 17 host IP=10.1.200.5 is enabled

SRPT093 AML: local side AML connection 17 to 10.1.200.5 health change:2.

SRPT091 HB: Local side AML health change:4.

SRPT043 LCS: Protected memory and disk synchronization complete.

SRPT050 DR: Disk sync completed.

SRPT096 HIRM: cmdu 0 16 1 on the remote side 0 is In Service.

SRPT026 HB: Remote side health change: 26 .

SRPT092 AML: remote side AML connection 17 to 10.1.200.5 health change: 2.

SRPT091 HB: Remote side AML health change:4.

SRPT062 Request to register TN rejected. UNEQUIPPED tn = 196-00-14-12 hwid = 18001765f6c
2456602

CSA104 17, 8:05:02 27/08/2010

VTN001 17 4 0 8 0 8 5 2
CDN002 17 9799 08 05 04

DTC001



TTY 05 SCH MTC BUG 8:07
OVL111 IDLE 0
>$$

 

[Tman45]

Lots from the INI messages - I have never seen so many output Your first one is the INI 2 I/O device fault - there are users on this forum that can point you in the right direction based on the hex code too (not me). Based on the INI 9 and INI 10 messages is does seem to be a network PE/MGS related issue. I know we have replaced PE cards before and others before

INI0009 xx Network faults seen from the standby CPU, where xx is the maintenance
display
code (see HEX). This is a minor alarm.
This message only appears if the standby CPU may be used but with a
degradation of network or I/O access as shown by comparing INI0002 with
INI0008, INI0003 with INI0009, and INI0007 with INI0010.
Action:
Severity: Critical. Critical to Monitor: Yes. SNMP trap: Yes

INI0010 xx IGS or MGS faults seen from the standby CPU, where xx is the maintenance
display code in HEX.
This message only appears if the standby CPU may be used but with a
degradation of network or I/O access as shown by comparing INI0002 with
INI0008, INI0003 with INI0009, and INI0007 with INI0010.
Action:
Severity: Critical. Critical to Monitor: Yes. SNMP trap: Yes
INI0007 xx Intergroup Switch (IGS) pack fault, where xx is the maintenance display code
in HEX.
Action:
Severity: Critical. Critical to Monitor: Yes. SNMP trap: Yes

INI0002xx I/O device fault, where xx is to the maintenance display code in.
Action:
Severity: Critical. Critical to Monitor: Yes. SNMP trap: Yes

INI0003 xx Network or PE signaling fault, where xx is the maintenance display code in
HEX.
Action:
Severity: Critical. Critical to Monitor: Yes. SNMP trap: Yes

 

[GHTROUT]

Anything that INIs 3 times per day, needs a reload



--
GHTROUT.com - Help for Nortel Meridian/CS1000 System Administrators
--

 

[TelcoChic]

@GHTROUT- LOL- that does not help!

Again, only INIs during the day!

 

[GHTROUT]

Take a look at the modems connected to the system, and make sure they are all set to "dumb mode"

A lot of INIs is often TTY/SDI related. In other words, data coming in where it is expected to go out. A modem set to 'smart mode' or verbal results can feed junk back onto the serial port.

A test would be to unplug all the ADAN TTYs and see if the system does not INI that day.

Something does not occur at night and on weekends, and those include TTY I/O, ACD Reports, busy CDR, - all TTY related.



--
GHTROUT.com - Help for Nortel Meridian/CS1000 System Administrators
--

 

[Triton101]

Question, this capture that you provided, is that directly capture from a TTY on and SDI card / Paddle board? or is the TTY on the CPSI?

You might be having a System realod issue not an INI issue, if you are capturing directly on a TTY.

Prt the AHST in ld 22 to see if you get the SYSLOAD message, that might provide more info.

__________________________________________________________
Find a job you love and you'll never work a day in your life. - Confucius

 

[johnpoole]

it happens during the day, so it would seem to be load dependant.. your overloading either speech paths, ram.. something is way out of kilter..

i've seen 30 stations with the same scr key do that. i've seen one that was only seeing one stick of ram do that.. i would look close at midn reports, pdt will help a lot faster if you have that access..

a relaod would not hurt, but it may not solve the problem either..

john poole
bellsouth business
columbia,sc

 

[TelcoChic]

So after trying everything under the sun- I opened a ETAS ticket this morning. So far- they dont have a clue either! I will let everyone know what the outcome is

Thanks for all the input- and references!

 

[mastang13]

Is the switch properly grounded? I've heard of some weird stories that had to do with grounding issues. Just a thought.

 

[johnpoole]

/u/patch/p28445_1.pp4".


ini during high traffic

john poole
bellsouth business
columbia,sc

 

[trvlr1]

Are you seeing any errors on your circuits, specifically the ones providing timing? If the count gets bad enough, I have seen this issue INI the PBX.
Often you will only see this problem when traffic builds up on the system, ie during the day, but at night and week ends when there is little or no traffic on the system/circuits it does not build up many errors and never INI's.
Check your counters in LD 60

 

[TelcoChic]

@mastang13 - we already had the grounding and rectifiers tested and checked.

@trvlr1 - the circuits are running fine and clean

@johnpoole- remember this is DSN, so the patch needs to end in .atv or I cant load it!

@everyone- Nortel still has the ticket- not much fun working these days.

 

[TelcoChic]

Resolution!!!

First : Further investigation found that dep list patch 20911 was causing the INI. The patch library shows that patch is obsolete (but still in the active DSN dep list!!) and replaced with 21935. Removed patch 20911 and replaced with 21935

Second: Found out the patch 20911 was performing incorrect procedure, patch rewritten to correct procedure and reloaded.

System has been running SMOOTHLY ever since!!!