OSB X8 v2 restarting

[plokijuh]

Hi guys,

I have an issue with OSB X8 v2.

Here is the part of evtlog.log file.

I get a lot of "!Poolerror" after that OSB X8 v2 goes to restart.

I've already changed the firmware to R6.2.0_029 (which is the last at the moment) with no results.

What do I need to check first?

Thanks.



...
EventLogEntry from FP_RECOVERY [fp:192.168.1.251] (0xb4614450 "09/27/2019 05:02:00.443251" E 0):
EventType: Indeterminate
EventCode: FP_EVT_GEN_016
EventText: 0xff C #16/16 0xff-0xff !Poolerror: 7e0d0053484f30e106b6
EventLogEntry from FP_RECOVERY [fp:192.168.1.251] (0xb4614450 "09/27/2019 05:02:00.445579" E 0):
EventType: Indeterminate
EventCode: FP_EVT_GEN_016
EventText: 0xff C #16/16 0xff-0xff !Poolerror: 032c0300bf823101019b
EventLogEntry from FP_RECOVERY [fp:192.168.1.251] (0xb4614450 "09/27/2019 05:02:00.447556" E 0):
EventType: Indeterminate
EventCode: FP_EVT_GEN_016
EventText: 0xff C #16/16 0xff-0xff !Poolerror: 03250600100d01000022
EventLogEntry from FP_RECOVERY [fp:192.168.1.251] (0xb4614450 "09/27/2019 05:02:00.449223" E 0):
EventType: Indeterminate
EventCode: FP_EVT_GEN_016
EventText: 0xff C #16/16 0xff-0xff !Poolerror: 0400004400a00e380079
EventLogEntry from FP_RECOVERY [fp:192.168.1.251] (0xb4614450 "09/27/2019 05:02:00.450952" E 0):
EventType: Indeterminate
EventCode: FP_EVT_GEN_016
EventText: 0xff C #16/16 0xff-0xff !Poolerror: 7f7f7f7f7f7f7f7f8104
EventLogEntry from FP_RECOVERY [fp:192.168.1.251] (0xb4614450 "09/27/2019 05:02:00.452830" E 0):
EventType: Indeterminate
EventCode: FP_EVT_GEN_016
EventText: 0xff C #16/16 0xff-0xff !Poolerror: 00c500000c000088804f
EventLogEntry from FP_RECOVERY [fp:192.168.1.251] (0xb4614450 "09/27/2019 05:02:00.454590" E 0):
EventType: Indeterminate
EventCode: FP_EVT_GEN_016
EventText: 0xff C #16/16 0xff-0xff !Poolerror: 020000ffffffffffffff
EventLogEntry from FP_RECOVERY [fp:192.168.1.251] (0xb4614450 "09/27/2019 05:02:00.844278" E 0):
EventType: Indeterminate
EventCode: FP_EVT_NW_000
EventText: 0x79 C #23/0 0xff-0xff NW rel chn for rm: Chn:704 still connected
EventLogEntry from FP_RECOVERY [fp:192.168.1.251] (0xb4614450 "09/27/2019 05:02:00.844769" E 0):
EventType: Indeterminate
EventCode: FP_EVT_NW_000
EventText: 0x79 C #23/0 0xff-0xff Speech conn 192 disconnected l_port 1718
EventLogEntry from FP_RECOVERY [fp:192.168.1.251] (0xb3e14450 "09/27/2019 05:02:01.626447" E 0):
EventType: Minor
EventCode: FP_EVT_GEN_007
EventText: 0xff B #16/7 0xff-0xff !General OSF error: 00010046010203000000
EventLogEntry from FP_RECOVERY [fp:192.168.1.251] (0xb3e14450 "09/27/2019 05:02:01.627225" E 0):
EventType: Minor
EventCode: FP_EVT_GEN_007
EventText: 0xff B #16/7 0xff-0xff !this event stopped the trace
EventLogEntry from DISPIPC [ldh:192.168.1.251] (0xa6bfe450 "09/27/2019 05:02:02.684786" dispipc_tcpserver.cpp 1981):
EventType: Critical
EventCode: MSG_DISPIPC_CONNECTION_CLOSED
EventText: Connection to DispIpc client tTCPServer::[192.168.1.251]:35640 closed

##### Startup Load osbiz_v2_R6.2.0_029 (LDH: M-LDH2.60.027, FP: M-FPX2.60.027b, RootFS: )
at "09/27/2019 05:05:12.021246" on 192.168.1.251 (last reboot: , PowerOn) #####

EventLogEntry from KERNEL [ldh:192.168.1.251] (0xb5dff450 "09/27/2019 05:05:13.050877" cnamedpipecollthread.cpp 88):
EventType: Warning
EventCode: MSG_OAM_INTERNAL_EVENT
EventText: Trace/Event Collector: start X=2 S=1 times
EventLogEntry from DYNDNS_NRT [ldh:192.168.1.251] (0xb5dff450 "09/27/2019 05:05:25.711433" ipsec_admin_recv.cpp 571):
EventType: Information
EventCode: MSG_IPSEC_INFORMATION
EventText: *** NRT-fSendAndGetMsg: IPSTACK_GLOBAL- Invalid DNS Server = 0.0.0.0
EventLogEntry from FP_RECOVERY [fp:192.168.1.251] (0xb5214450 "09/27/2019 05:05:34.358089" E 0):
EventType: Cleared
EventCode: FP_EVT_GEN_011
EventText: 0x61 B #16/11 0xff-0xff RESTARTED: Application reset
EventLogEntry from CUSTOMER_TRACE [ldh:192.168.1.251] (0xb31ff450 "09/27/2019 05:05:34.361781" ccustomertrcsvc01.cpp 200):
EventType: Information
EventCode: MSG_CUSTOMER_OR_SERVICE_EVENT
EventText: 27 Sep 2019 05:05:34:36 System :1:
System restart because of OBSERVER.
...

 

[plokijuh]

Any thoughts guys?

 

[adibv]

Check if you have an error in LCR, or with a forwarding on the phone.
This could potentially be a loop where a call gets routed out of the system and back in, back out, back in, etc.
Disconnect either LAN or WAN network cableand chech if reboots stop and these errors gone.

 

[adibv]

On Openscape business V2 feature description and Administratir documentation service manuals:
On Chapter 25.8.2 SNMP (Simple Network Management Protocol)
EventCode: FP_EVT_GEN_007 means :
General OSF error
For EventCode: FP_EVT_GEN_016 and EventCode: FP_EVT_NW_000 there is no info.

 

[plokijuh]

Thank you for the answer adibv.

I did check all of the LCR but there wasn't anything really noticeable.

Anyway I deleted all unnecessary entries.

adibv can any error in LCR or forwarding on the phone lead OSB X8 to restart?

 

[adibv]

Yes, a network loop can lead to a restart.
But if you are sure that there is no config error, I think a deeper investigation (by Unify ?) is needed since I cannot fully decode the logs.

 

[plokijuh]

Thank you very much for replies adibv.

The problem is still exists. But after some changes to Trunks (clearing unnecessary entries) and LCR (clearing unnecessary entries) it is already 7 days without any restart.

 

[t3rm3y]

If can't find a reason for a fault or it reappears might be worth a factory reset and rebuild. Annoying but can rebuild system .
Was it a new osbiz or hipath upgrade?
If upgrade were the existing cards check that they are compatible?

 

[plokijuh]

Thank you for reply t3rm3y.

We already did a factory reset but used saved KDS for configure. And we've changed SD Card.
All cards are compatible.

OSB is still restarting time to time though.
At the moment network guys checking the network. We'll see what this leads to.

 

[donb01]

I would associate "pool" with memory or other system resources. Maybe try reseating the processor or the memory on the processor? I'm just throwing this out there blindly so it may not be relevant. I have had memory issues in the 9006 days....

Don Bruechert, Voice Comm Analyst II
CareTech Solutions @ Holy Family Memorial
Manitowoc, WI, USA

 

[plokijuh]

Thank you for reply donb01.

Yes, indeed, it looks like a memory pool error and there are memory addresses in the log file for every pool error record.

As I mentioned before, I've already cleared all unnecessary LCR and Trunk records from the OSB X8 and I had 10 days without any restart.
But after 10 days, OSB X8 started to reboot again.

I've checked cisco port which was connected to OSB X8 and it had a lot off CRC errors which may couse port or cable issue at the first glance.
At the moment I changed network cable.
It's about 3 or 4 days without restarts after changing network cable.
We kept the same cisco port though.

 

[donb01]

Being a telecom guy, most times when I have reboot issues with my phones that are bad enough that the customer notices and complains the first thing I do is blame the phone end patch cord (75% of the time that's accurate), then the patch cord in the rack (maybe 5% of the time) and then the drop, which usually isn't the cause unless something was touched in the area. Usually it's not the drop. The rest of the time I replace the phone because I dodn't know what else to do. So you might get lucky with replacing the patch cord!

I don't know how the memory is allocated on that system, but other possible consumers of memory are big speed dial lists, or having speed dial lists assigned to phones that don't need them, CDR buffers that are storing data when you don't use CDR (that's more a disk issue), and a few other things I can't think of right now.

Here's hoping it stays good!

Don Bruechert, Voice Comm Analyst II
CareTech Solutions @ Holy Family Memorial
Manitowoc, WI, USA

 

[thepfy]

Is the power feed stable? (via UPS?)

If there are CRC errors on the Cisco Network Port, the first thing I would check is for speed and duplex mismatches between the X8 and the Cisco Port.

A stuck call could be a possibility. The old ISDX would do a switch between the processors if it had a call it didn't know what to do with (often routing errors or when someone takes down a section of LAN with many IP phones...). Most modern systems just drop the call though.
A looping call could be an issue, making the system run out of memory.

 

[plokijuh]

Hi guys.

Changing network cable didn't help. 4 days without restart and then boom.

In my opinion the problem is not the OSB X8. There is some issues in the network.

At the moment network admins analyzing network.

After changing network cable CRC error gone though.

thepfy thank you for reply.
I have UPS and it is ok. We have no issues with the power.
speed and duplex is ok on both side.
It seems that memory is leaking but memory is ok on the OSB X8. Also I've changed the SD Card.

I think Network Time has come.

 

[plokijuh]

Hey guys.

Did update OSB X8 to newer version V2 R7.0.0 871 with no effect. OSB X8 is still restarting.

I think it is network problem but cannot determine what can be cause.

What if I turn on Firewall on OSB X8?

 

[xs650]

Open a ticket at Unify and attach the diagnostic data with activated corelog protocols.

 

[plokijuh]

Hi guys.

So far we've decided to use DIUN2 board and unplug OSB X8 from the network.

Thanks guys for the help.