OpenScape 4000 Standby goes to restart

[plokijuh]

Hi guys,

have an issue with OpenScape 4000 v8 Duplex.

I have no problem with the Active, but Standby goes to resatrt 10-15 times a day.

When restart occurs this errors appears



F4057 M8 N1741 NO ACT BPA CP DBAR 19-09-19 20:09:25
ALARM CLASS:CENTRAL:023
CC:00566 EC:00506 UA:BAF0:C729 SP:4A8C:17C8 LD:01-18-005-009
DT:6C ST:6F SN: 1 CEVT:2A CSEV:6C CST: 0
FORMAT:24 MESSAGE-ID: 10248

A9012 M8 N1742 NO ACT A1 NMCALARM PER-MIRR: CENTRAL GRP 19-09-19 20:09:57
DEVICE ALARM: 0000000000000000
MINOR ALARM: 0000000000000000
MAJOR ALARM: 0000000000000000
FORMAT:30

A9015 M8 N1743 NO ACT A1 NMCALARM PER-MIRR: SMPER GRP 19-09-19 20:09:57
DEVICE ALARM: 000000
MINOR ALARM: 000000
MAJOR ALARM: 000000
FORMAT:30

A9012 M8 N1744 NO ACT BPA NMCALARM PER-MIRR: CENTRAL GRP 19-09-19 20:09:57
DEVICE ALARM: 0000000000000000
MINOR ALARM: 2000002000000000
MAJOR ALARM: 2000002000000000
FORMAT:30

A9013 M8 N1745 NO ACT BPA NMCALARM PER-MIRR: SWUPER GRP 19-09-19 20:09:57
DEVICE ALARM: 00000000
MINOR ALARM: 00000000
MAJOR ALARM: 00000000
FORMAT:30

A9014 M8 N1746 NO ACT BPA NMCALARM PER-MIRR: SWULOG GRP 19-09-19 20:09:57
DEVICE ALARM: 000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000
MINOR ALARM: 000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000
MAJOR ALARM: 000000000000000000000000000000000000000000000000000000000000
000000000000000000000000000000000000000000000000000000000000
00000000000000000000000000
FORMAT:30

F4057 M8 N1751 NO ACT BPA CP DBAR 19-09-19 20:10:21
ALARM CLASS:CENTRAL:023
CC:00566 EC:00506 UA:BAF0:C729 SP:4A8C:17C8 LD:01-18-005-009
DT:6C ST:6F SN: 1 CEVT:2A CSEV:6C CST: 0
FORMAT:24 MESSAGE-ID: 10249

6C6F0100 2A6CDF08 00000000 00FF2806 00000002 01000100 2814000E 0400046F
01000500 00000000 00000000 00000033 05020000 0AC00AC0 6B036B03 00000000
15000000 01140325 00000102 00006B03 DF08FFFF FFFFFFFF FFFF0000 2C0101



I will be grateful for any help.

 

[adibv]

Hi.

F4057 err mess in Hista UA:BAF0:C729 means (for 4k V8 R2 sys ver):
VFGR is missing on the system and needs to be added

Cannot tell if this leads to STDBY CC restart.

Check ATND config ,WABE (ATND,ATNDDID,ATNDIND),VFGR,VFGKZ entries.

codes for ATND and ATNDDID must be configured on AMO WABE and VFGKZ and it has to be assigned to an attendant group.
NOTE: this is even if the ATNDDID code will not be used by the user.

Check also for F4066 err mess with CC:15341

 

[plokijuh]

Thank you for reply adibv. Will check it and reply.

 

[george_a]

These messages might be related, but they are not showing the cause of the restart, you need to take a larger portion of the Hista around the time of the restart, usually the relevant messages are marked accordingly with something like HARDREST or SOFTREST right before the name of the SWU processor - BPA or BPB.

 

[plokijuh]

Thank you for reply george_a.

DISPLAY-HISTA:TAB=HWES,TTYPE=ALARMS;
gives me a lot of F8266 errors.

DISPLAY-HISTA:TAB=SWES;
gives me only F4057.

I only look at recent errors. Within a week.

 

[Moriendi]

If it's happening 10-15 times a day, you have lots of opportunity to get the hista.

sta-hista:search; will give you the last two hours of hista.

Wait until it makes another restart, do the sta-hista as above, and post that output. Don't try to filter it because you don't know what's relevant and what's not.

 

[eniyamath]

Hello

Actually if the standby server unable to synchronize with the active server, it tries to restart by itself to synchronize .

During installation if the DRDB synchronize properly this types of restart problem wont rise.

Check the DRDB status in assistant

 

[Moriendi]

I would expect ATLAN problems to cause standby restart, not DRBD. DRBD is for the clustering needed at the Linux level. The three RMX processors ADP CCA CCB are all communicating over ATLAN, not DRBD.

HISTA will reveal.

 

[plokijuh]

Thank you Moriendi and eniyamath for replies.

As adibv advised I need to add these commands:
ADD-NAVAR:1,NONE;
ADD-VFGR:0,DQ,,1,,,,,,,,1,,20,0&1&2&3&4&5&6&7&8&9&10&11&12&13&14&15,OFF,NO,YES,NO,,0,OFF,NO;
CHANGE-ZANDE:ATND,NO,NO;
CHANGE-ZAND:ATND,2,EXT1,YES,YES,NO,NO,NO,NO,NO,NO,NO,NO,NO,NO,NO,0,NO,NO,NO,NO,1,NO,NO,NO,NO,YES;
CHANGE-CTIME:ATND,60,2,5,20,15,30,7;
At the moment I do not have these on PBX. At least I can get rid of part of errors.



Here is the restart time and part of HISTA. Thank you.

<DISPLAY-SIGNL:SYSTEM;
DISPLAY-SIGNL:SYSTEM;
H500: AMO SIGNL STARTED
+------------------------------------------------------------------------------+
| SYSTEM STATE TABLE |
+----------+-------------------------+-----------------------------------------+
| UNIT | STATE | ADDITIONAL INFORMATION |
+----------+-------------------------+-----------------------------------------+
| CC-A | ACTIVE | RELOAD FROM 19-07-18 17:46:12 |
| CC-B | STANDBY | RELOAD FROM 19-09-25 17:44:59 |
| ADS | IN SERVICE | SOFT RESTART FROM 19-07-30 10:24:57 |
| REF CLOCK| AUTONOMOUS CLOCK | |
| | (NO CLOCK REFERENCE) | |
| TFAILTR | TRUNK FAIL TRANSFER OFF | |
| ALARM | DEVICE ALARM | ---- ---- |
| | MINOR ALARM | IN SWU ---- |
| | MAJOR ALARM | IN SWU ---- |
+----------+-------------------------+-----------------------------------------+



START-HISTA:RTYPE=SEARCH,STIME="2019-09-25/17:40",ETIME="2019-09-25/17:50";

F4057 M8 N7242 NO ACT BPA CP DBAR 19-09-25 17:43:48
ALARM CLASS:CENTRAL:023
CC:00566 EC:00506 UA:BAF0:C729 SP:4A8C:17CE LD:00-00-000-000
DT:6C ST:6C SN:5401 CEVT:EF CSEV: 0 CST: 0
FORMAT:24 MESSAGE-ID: 16455

FF030101 88010102 04050700 09080000 80000000 00000001 00020000 00000000
00000000 03000000 00000002 E0020006 058C4A04 00000020 00000000 00000000
00000000 00000000 00000000 00000000 00000804 03070000 06000006 06030000
00400000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0C000000 00000F0F 0F0F0F0F 00020104 010A0A15 25000D00 00007A03 0825AC0A
011260

F4057 M8 N7243 NO ACT BPA CP DBAR 19-09-25 17:43:48
ALARM CLASS:CENTRAL:023
CC:00566 EC:00506 UA:BAF0:C729 SP:4A8C:17CE LD:00-00-000-000
DT:6C ST:6C SN:5401 CEVT:EF CSEV: 0 CST: 0
FORMAT:24 MESSAGE-ID: 16455

F8266 M4 N7244 NO ACT A1 TRANSSYS ON CONN LOSS OS 19-09-25 17:44:11
ALARM CLASS:CENTRAL:023
PARTNER:BPB IP-ADDR:192.0.2.2 TCP-PORT:640
LOCAL: IP-ADDR:192.0.2.3 TCP-PORT:0
** : :A1 B :062: K7754-X ECOSRV BST:01 FW: H062-Q
FORMAT:3C
SUB-MSG-NO:1

F8266 M4 N7245 LOCK BPA TRANSSYS ON CONN LOSS OS 19-09-25 17:44:13
ALARM CLASS:CENTRAL:023
PARTNER:BPB IP-ADDR:192.0.2.2 TCP-PORT:641
LOCAL: IP-ADDR:192.0.2.1 TCP-PORT:0
P202: :CC A :002: ** H'007F BST:02 FW: H062-Q
FORMAT:3C
SUB-MSG-NO:1

F8267 M4 N7246 IN SERV BPA TRANSSYS ON CONN OS NEW 19-09-25 17:44:14
ALARM CLASS:CENTRAL:023
PARTNER:BPB IP-ADDR:192.0.2.2 TCP-PORT:0
LOCAL: IP-ADDR:192.0.2.1 TCP-PORT:0
P202: :CC A :002: ** H'007F BST:02 FW: H062-Q
FORMAT:3C
SUB-MSG-NO:1

F4266 M8 N7247 NO ACT BPB DEP ADVISORY 19-09-25 17:43:44
ALARM CLASS:CENTRAL:005
CC:65535 EC:65535 UA:FFFF:FFFF SP:FFFF:FFFF LD:00-00-000-000
DT:FF ST:FF SN:FFFF CEVT:FF CSEV:FF CST:FF
FORMAT:24

4034133B 3A000A50 30343535 32204445 563A4641 53202020 54534E3A 36393931
20554944 3A303320 4E4F554E 3A534947 4E4C2054 5950453A 53544152 54202020
2054494D 453A3139 2D30392D 32352031 373A3433 3A34330A 14004449 532D5349
B800005A 5A5A5A5A 5A5A5A5A 5A5A5A5A 5A5A5A5A 5A5A5A5A 5A5A5A5A 5A5A5A5A
5A5A5A5A 5A5A5A5A 5A5A5A5A 5A5A5A5A 5A5A5A5A 5A5A5A5A 5A5A5A5A 5A5A5A5A
5A5A5A5A 5A5A5A5A 5A5A5A5A 5A5A5A5A 5A5A5A5A 5A5A5A5A 5A5A5A5A 5A5A5A5A
5A5A5A5A 5A5A5A5A

F4266 M8 N7248 NO ACT BPB DEP ADVISORY 19-09-25 17:44:11
ALARM CLASS:CENTRAL:005
CC:00566 EC:00506 UA:BAF0:C729 SP:8A04:0704 BP:071E LD:00-00-000-000
FORMAT:45 MESSAGE-ID: 05318
STACK-DATA-MESSAGE 01 OF 02
---4---6 ---8---A ---C---E ---0---2 ---4---6 ---8---A ---C---E ---0---2
48A35308 4E680726 6D2A0ABC 00000000 0000087B 20080000 64000728 DF98BAF8
01880188 077466E1 0ABC0188 074605F8 087B1CD4 00039C04 0AC4E5D7 BAC00788
007C077C 05F8087B 1CD40003 9C040816 0ABC0788 07642495 077205F8 48A35308
077207CC 62EB0ABC 086C0AC4 030307CC 07CC07CC 0AC489D7 0ABC0002 9CAC2008
0B9F0AC4 087B0AC4 00520002 9CAC0002 9CAC00DE 00001178 024607BA 0921C42D
132007B8 8A040002 9CAC0C00 00000001 9CAC0000 000007DA 01880000 132007D8

F4266 M8 N7249 NO ACT BPB DEP ADVISORY 19-09-25 17:44:11
ALARM CLASS:CENTRAL:005
CC:00566 EC:00506 UA:BAF0:C729 SP:8A04:07C4 BP:071E LD:00-00-000-000
FORMAT:45 MESSAGE-ID: 05318
STACK-DATA-MESSAGE 02 OF 02
---4---6 ---8---A ---C---E ---0---2 ---4---6 ---8---A ---C---E ---0---2
8A041064 01000AC4 07DC07DC 0AC43100 0ABC0002 9CAC0200 07F45A5A 0AC43283
0ABC0002 9CAC0000 000B0AC4 00000000 00000000 FFFF8A0C 31F70ABC

F4259 M8 N7250 SOFTREST BPB DEP DYNAMIC DATA FAULT 19-09-25 17:44:11
ALARM CLASS:CENTRAL:005
CC:00566 EC:00506 UA:BAF0:C729 SP:8A04:0704 LD:00-00-000-000
DT:5A ST:6C SN:5401 CEVT: 1 CSEV:FF CST: 2
FORMAT:24 MESSAGE-ID: 05318

5A6C0154 0103002C 00000888 07001615 73880100 00010400 00000000 01011203
00008000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 73940000 0000020B 00040000 00001700 07BE0400
04000000 00170009 BE040000 00000000 00002080 00000000 20800000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 0000007B EA02EA7B FB041600 00000000 00000000
00000000 00000000

F4266 M8 N7251 HREXSRRE BPB DEP ADVISORY 19-09-25 17:44:11
ALARM CLASS:CENTRAL:005
FORMAT:42
SR DENIED:
CC IS STANDBY OR
RESTART ALREADY RUNNING

F4706 E8 N7252 HRCCTSS BPB DEP RESTART LEVEL 19-09-25 17:44:14
ALARM CLASS:CENTRAL:005
RESTART TYPES: HARD RESTART

FORMAT:41

F8267 M4 N7253 NO ACT A1 TRANSSYS ON CONN OS NEW 19-09-25 17:44:14
ALARM CLASS:CENTRAL:023
PARTNER:BPB IP-ADDR:192.0.2.2 TCP-PORT:0
LOCAL: IP-ADDR:192.0.2.3 TCP-PORT:0
** : :A1 B :062: K7754-X ECOSRV BST:01 FW: H062-Q
FORMAT:3C
SUB-MSG-NO:1

F6016 M8 N7254 STATIST BPB DMS ADVISORY 19-09-25 17:44:16
ALARM CLASS:CENTRAL:023
FORMAT:49
BOOT/SYSLOAD INFOFILE NO WRITE - CHECK HHD

F6016 M8 N7255 STATIST BPB DMS ADVISORY 19-09-25 17:44:48
ALARM CLASS:CENTRAL:023
FORMAT:49
BOOT/SYSLOAD INFOFILE WRITTEN SUCCESSFULLY

F3177 M4 N7256 NO ACT BPB SWU-INT RTM RESET 19-09-25 17:44:50
ALARM CLASS:CENTRAL:017
FORMAT:0

F4604 E8 N7257 NO ACT BPB DB MAINTENANCE 19-09-25 17:44:50
ALARM CLASS:CENTRAL:029
FORMAT:33
AUTO TRACER: BUFFER 03827 STORED ON HD. RESTART TIME: 17:44:11

F4057 M8 N7261 NO ACT BPA CP DBAR 19-09-25 17:45:01
ALARM CLASS:CENTRAL:023
CC:00566 EC:00506 UA:BAF0:C729 SP:4A8C:1858 LD:01-01-012-000
DT:6C ST:6F SN: 1 CEVT:CD CSEV:3B CST: 0
FORMAT:24 MESSAGE-ID: 16466

6C6F0100 CD3B8807 00180000 D6000008 0000FD04 1C2301D0 119E049C EA01008B
0100A122 0C400103 00200000 00060804 05150000 1CA01CA0 8C018C01 8C010000
15000000 01B10249 00004A01 00008C01 8807FFFF FFFFFFFF FFFF0000 2C0101

F4057 M8 N7262 NO ACT BPA CP DBAR 19-09-25 17:45:01
ALARM CLASS:CENTRAL:023
CC:00566 EC:00506 UA:BAF0:C729 SP:4A8C:1858 LD:01-01-012-000
DT:6C ST:6F SN: 1 CEVT:CD CSEV:3B CST: 0
FORMAT:24 MESSAGE-ID: 16466

FF020115 1CA00300 00538C01 00B63866 1CA01CA0 88070018 8C018C01 FFFFFFFF
00000000 0002C000 00800000 00000000 2C3BCD08 C0000116 03000080 00FFFF00
00FFFF00 00FFFF00 00FFFF00 00FFFF00 00220000 00000000 00000000 00000000
00000000 00000000 004B9B8D 03000000 00000000 00491500 6640B428 66B0188C
4A88BB83 30060001 C014

F4057 M8 N7263 NO ACT BPA CP DBAR 19-09-25 17:45:01
ALARM CLASS:CENTRAL:023
CC:00566 EC:00506 UA:BAF0:C729 SP:4A8C:1858 LD:01-01-012-000
DT:6C ST:6F SN: 1 CEVT:CD CSEV:3B CST: 0
FORMAT:24 MESSAGE-ID: 16466

FF030101 8C010501 05020902 09080000 80000000 00000001 00020000 00000000
00000000 03000000 00000000 000F0F0F 0F0F0F00 00000020 00000000 00000000
00000000 00000000 00000000 00000000 00000804 03070000 06000006 00030000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
0C000000 00000F0F 0F0F0F0F 00020104 010A0ABF 04FFB6BB 305180BB 3051FFFF
011CA0

F4057 M8 N7264 NO ACT BPA CP DBAR 19-09-25 17:45:01
ALARM CLASS:CENTRAL:023
CC:00566 EC:00506 UA:BAF0:C729 SP:4A8C:1858 LD:01-01-012-000
DT:6C ST:6F SN: 1 CEVT:CD CSEV:3B CST: 0
FORMAT:24 MESSAGE-ID: 16466

F4057 M8 N7269 NO ACT BPA CP DBAR 19-09-25 17:45:01
ALARM CLASS:CENTRAL:023
CC:00566 EC:00506 UA:BAF0:C729 SP:4A8C:17CE LD:00-00-000-000
DT:6C ST:6C SN:B601 CEVT:EF CSEV: 0 CST: 0
FORMAT:24 MESSAGE-ID: 16467

6C6C01B6 EF00A606 00000000 88070018 8C0102E0 1CA00000 00000000 00000000
00000000 00000003 48113200 04130100 05490000 1CA01CA0 360102E0 00003601
00000000 02090082 00004A01 00003601 A606FFFF FFFFFFFF FFFF0000 2C0101

F4057 M8 N7270 NO ACT BPA CP DBAR 19-09-25 17:45:01
ALARM CLASS:CENTRAL:023
CC:00566 EC:00506 UA:BAF0:C729 SP:4A8C:17CE LD:00-00-000-000
DT:6C ST:6C SN:B601 CEVT:EF CSEV: 0 CST: 0
FORMAT:24 MESSAGE-ID: 16467

FF020149 1CA00300 005302E0 8003986B 1CA01CA0 A60602E0 02E03601 88070018
8C010000 00000900 00800000 00040000 2C00EF10 40000000 80000080 00FFFF00
00FFFF00 00FFFF00 00FFFF00 00FFFF00 00000000 00000000 00000000 00000000
00000000 00000000 00000000 00000000 00000000 00820000 09000080 000000F4
F4F4F400 FF0F0000 0000

 

[donb01]

I'm gonna be a PBX guy and blame that on a bad network cable, or data drop, or patch cord in the rack. It is saying it it is rebooting on loss of connection. I have lots of phones that say "loss of layer 1 signalling" all the time, and it's almost always one of the network cables for that.


Don Bruechert, Voice Comm Analyst II
CareTech Solutions @ Holy Family Memorial
Manitowoc, WI, USA

 

[Moriendi]

No it lost the connection because it rebooted. ADP and BPA both report loss of connection to BPB, but that's because BPB rebooted.

I would say the cause of restart is this "F4259 M8 N7250 SOFTREST BPB DEP DYNAMIC DATA FAULT 19-09-25 17:44:11" but this is where it pays to have a maintenance contract. The 4K has written autotrace files which should be investigated by Unify, and if they don't help, a proper trace can be put on the 4K to capture the next restart. Although all those DBARs happening around the same time sure are interesting. I would also be looking to see if you get the restart without the DBARs, if not, fix the DBARs.

I'd also make sure the 4K is on the latest patch version.

 

[plokijuh]

Thank you for replies donb01 and Moriendi.

I will check the cables. But I didn't have any issue during installation process.

4K is on the latest patch version.

DBARs - F4057 as mentioned before adibv can be resolved by adding VFGR to the system.
At the moment VFGR is missing.
I planning to run these commands:
ADD-NAVAR:1,NONE;
ADD-VFGR:0,DQ,,1,,,,,,,,1,,20,0&1&2&3&4&5&6&7&8&9&10&11&12&13&14&15,OFF,NO,YES,NO,,0,OFF,NO;
CHANGE-ZANDE:ATND,NO,NO;
CHANGE-ZAND:ATND,2,EXT1,YES,YES,NO,NO,NO,NO,NO,NO,NO,NO,NO,NO,NO,0,NO,NO,NO,NO,1,NO,NO,NO,NO,YES;
CHANGE-CTIME:ATND,60,2,5,20,15,30,7;

I'm planning to do it tomorrow and will write the result.

 

[george_a]

The standby CC restarts are indeed related to the missing attendant group configuration, the user address is the same in the advisories.
On the active side you only get a DBAR as CP tries to handle a call for which interception to attendant group would make sense, but on the standby side when such a call's related data has to be saved by the Recovery mechanism, the exception handler's reaction is to trigger a soft restart, which always escalates to a hard restart for a standby CC.

active side:

F4057 M8 N7243 NO ACT BPA CP DBAR 19-09-25 17:43:48
ALARM CLASS:CENTRAL:023
CC:00566 EC:00506 UA:BAF0:C729 SP:4A8C:17CE LD:00-00-000-000
DT:6C ST:6C SN:5401 CEVT:EF CSEV: 0 CST: 0
FORMAT:24 MESSAGE-ID: 16455


standby side:

F4259 M8 N7250 SOFTREST BPB DEP DYNAMIC DATA FAULT 19-09-25 17:44:11
ALARM CLASS:CENTRAL:005
CC:00566 EC:00506 UA:BAF0:C729 SP:8A04:0704 LD:00-00-000-000
DT:5A ST:6C SN:5401 CEVT: 1 CSEV:FF CST: 2
FORMAT:24 MESSAGE-ID: 05318

F4266 M8 N7251 HREXSRRE BPB DEP ADVISORY 19-09-25 17:44:11
ALARM CLASS:CENTRAL:005
FORMAT:42
SR DENIED:
CC IS STANDBY OR
RESTART ALREADY RUNNING

F4706 E8 N7252 HRCCTSS BPB DEP RESTART LEVEL 19-09-25 17:44:14
ALARM CLASS:CENTRAL:005
RESTART TYPES: HARD RESTART

FORMAT:41

 

[plokijuh]

Thank you george_a for reply.

I did changes.
ADD-NAVAR:1,NONE;
ADD-VFGR:0,DQ,,1,,,,,,,,1,,20,0&1&2&3&4&5&6&7&8&9&10&11&12&13&14&15,OFF,NO,YES,NO,,0,OFF,NO;
These two commands were enough because other commands were already in the config.

I didn't run EXEC-UPDAT:BP,ALL; though just in case to see what will happen.

After that I checked the HISTA.

START-HISTA:RTYPE=SEARCH,STIME="2019-09-27/18:15",ETIME="2019-09-27/21:00";
H500: AMO HISTA STARTED
H1: 00000 ERROR MESSAGES FOUND
AMO-HISTA-111 READ HTS HISTORY FILE
START COMPLETED;

As you can see all errors gone.

No restarts during this period.

I am going to check the PBX for several days and write results.

 

[plokijuh]

Thank you guys for the help.

It's already 4th day without restart.

F4057 error completely gone.