IP Office Server Edition 9.1 - SP6 - Sudden Crashing & Reboot 1

[superikey]

Hi All

Wondering if anyone running SE (Virtual on VM-Ware) 9.1 SP6 is having sudden crashing Issues.
This is a large multi-site call center deployment with a primary SE in Site #1 running VM Pro, 78 Voicemial Ports, w/125 Users ans 200 SIP Trunks, 40 remote users connected to site #1 using One-X + Communicator for Windows, a secondary SE in Site #2 with 80 users and 100 SIP Trunks, Xima Chornicall with Reporting, Real-Time and Recording Library.

System went Live Monday morning, and since Yesterday we have had 4-5 spontaneous crashes. We are working with Avaya Support/R&D, but wondering if anyone can shed some light or having similar issues.

Here are the Logs from Right before the crash..

13:41:56 76138317mS SIP Tx: UDP 192.168.30.5:5060 -> 66.234.226.20:5060
INVITE sip:14082700467@66.234.226.20 SIP/2.0
Via: SIP/2.0/UDP 192.168.30.5:5060;rport;branch=z9hG4bK7154acb7f095f04e54fd56e49d197a39
From: "Daniel Johns" <sip:7322435816@NYC1SIPTGID-5290>;tag=fa51b12a7d9bb828
To: <sip:14082700467@66.234.226.20>
Call-ID: fd1540ec2275484d94c2246a75033a3e
CSeq: 1550345014 INVITE
Contact: "Daniel Johns" <sip:7322435816@192.168.30.5:5060;transport=udp>
Proxy-Authorization: Digest username="9294006189",realm="NYC1SIPTGID-5290",nonce="5479828",response="1089ff6800e34343ce6fcfd146c89e55",uri="sip:14082700467@66.234.226.20"
Max-Forwards: 70
Allow: INVITE,ACK,CANCEL,OPTIONS,BYE,INFO,NOTIFY
Supported: timer
User-Agent: IP Office 9.1.6.0 build 153
Content-Type: application/sdp
Content-Length: 202

v=0
o=UserA 598183830 217318802 IN IP4 192.168.30.5
s=Session SDP
c=IN IP4 192.168.30.5
t=0 0
m=audio 46478 RTP/AVP 0 101
a=rtpmap:0 PCMU/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
13:41:56 76138317mS CD: CALL: 2.844.1 BState=Idle Cut=2 Music=0.0 Aend="Line 2" (252.9) Bend="" [Line 101] (0.0) CalledNum=514082700467 () CallingNum=8348 (Daniel Johns) Internal=0 Time=4292 AState=Dialling
13:41:56 76138326mS SIP Rx: UDP 66.234.226.20:5060 -> 192.168.30.5:5060
SIP/2.0 100 Trying
Via: SIP/2.0/UDP 192.168.30.5:5060;received=100.8.37.174;rport=5060;branch=z9hG4bK7154acb7f095f04e54fd56e49d197a39
To: <sip:14082700467@66.234.226.20>
From: "Daniel Johns" <sip:7322435816@NYC1SIPTGID-5290>;tag=fa51b12a7d9bb828
Call-ID: fd1540ec2275484d94c2246a75033a3e
CSeq: 1550345014 INVITE
Content-Length: 0

13:41:56 76138326mS CMLineRx: v=0
CMProceeding
Line: type=SIPLine 101 Call: lid=101 id=111015 in=0
Called[] Type=Default (100) Reason=CMDRdirect Calling[8348] Type=Internal Plan=Default
IE CMIERespondingPartyNumber (230)(P:100 S:100 T:0 N:100 R:4) number=514082700467
IE CMIEDeviceDetail (231) c0a80a050001b1a7 LOCALE=enu HW=11 VER=9 class=CMDeviceSIPTrunk type=0 number=101 channel=10 features=0x1 rx_gain=32 tx_gain=32 ep_callid=111015 ipaddr=192.168.10.5 apps=0 loc=999 em_loc=999 features2=0x0 is_spcall=1
13:41:56 76138326mS CMARS: LINE ep Received: CMProceeding - child->state = CMCSOffering - ARS Call State = CMCSOverlapRecv
13:41:56 76138340mS CMTARGET: c0a80a050001b173 10348.110963.0 19712 ES Melissa Reye.0: AttemptStopRinging
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Retarget NOANSWER EXCEPTED=00000000 INCLUDED=00000000 ValidTargets=0
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: RetargetNoAnswer on HUNTGROUP=Q New Claim. Ring Attempt 1
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: AddHGTarget Q New Claim (depth=1) allowq=0 type=CMNTypeDefault
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: AddHGTargetRingIdle(LongestWaiting) Q New Claim ring_attempt_count 1
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8352: Non Tapi User not logged in And FwdU Off
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8419: DND
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8654: Non Tapi User not logged in And FwdU Off
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8667: LII Uninitialised And FwdU Off
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8680: DND
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8685: LII Uninitialised And FwdU Off
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8647: LII Uninitialised And FwdU Off
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8678: LII Uninitialised And FwdU Off
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8672: LII Uninitialised And FwdU Off
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8676: LII Uninitialised And FwdU Off
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8624: Non Tapi User not logged in And FwdU Off
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8656: LII Uninitialised And FwdU Off
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8657: LII Uninitialised And FwdU Off
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8623: LII Uninitialised And FwdU Off
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8622: LII Uninitialised And FwdU Off
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8695: LII Uninitialised And FwdU Off
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8651: LII Uninitialised And FwdU Off
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8661: DND
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8628: LII Uninitialised And FwdU Off
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8644: LII Uninitialised And FwdU Off
13:41:56 76138340mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8681: LII Uninitialised And FwdU Off
13:41:56 76138341mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8664: LII Uninitialised And FwdU Off
13:41:56 76138341mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8675: DND
13:41:56 76138341mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8415: Non Tapi User not logged in And FwdU Off
13:41:56 76138341mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8418: Non Tapi User not logged in And FwdU Off
13:41:56 76138341mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8420: Non Tapi User not logged in And FwdU Off
13:41:56 76138341mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8423: Non Tapi User not logged in And FwdU Off
13:41:56 76138341mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: Q New Claim: Not using 8365: Non Tapi User not logged in And FwdU Off
13:41:56 76138341mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: ADD USER: ES Karen Marroq depth=2 disallow_cw=1 dnd=0 real_call=1 group_call=1 type(CMNTypeDefault) incl(0x0) excpt(0x3c), allow_redir(1) remote=00000000 simult 1 (0)
13:41:56 76138341mS CMCallEvt: 0000000000000000 0.111016.0 -1 BaseEP: NEW CMEndpoint d8a39e00 TOTAL NOW=467 CALL_LIST=203
13:41:56 76138341mS CMCallEvt: 0000000000000000 0.111016.0 -1 ES Karen Marroq.-1: NEW CMExtnEndpoint d8a39e00, Name=ES Karen Marroq, Extn=8626, Phys Extn=8626
13:41:56 76138341mS CMMap: IP::SetCodec pcp[858]b0r0 0 -> d9f9aa18
13:41:56 76138341mS CMTARGET: c0a80a050001b1a8 10315.111016.0 19712 ES Karen Marroq.0: ADD SIMULTANEOUS
13:41:56 76138342mS CMExtnEvt: ES Melissa Reye: CALL LOST (CMCauseRetargetNoAnswer)
13:41:56 76138342mS CMExtnEvt: ES Melissa Reye: Extn(8655) Calling Party Number(4045467300@24.170.214.10) Type(CMNTypeUnknown)
13:41:56 76138342mS CMCallEvt: c0a80a050001b173 10348.110963.0 -1 ES Melissa Reye.0: StateChange: END=X CMCSRinging->CMCSCompleted
13:41:56 76138342mS CMExtnEvt: v=0 State, new=PortRecoverDelay old=Ringing,0,0,ES Melissa Reye
13:41:56 76138342mS CMExtnEvt: ES Melissa Reye: CMExtnHandler::SetCurrent( id: 110963->0 )
13:41:56 76138342mS CMCallEvt: c0a80a050001b173 10348.110963.0 -1 ES Melissa Reye.-1: StateChange: END=X CMCSCompleted->CMCSDelete
13:41:56 76138342mS CMTARGET: CheckApplyPostAlertingStatus: Extn 8655 targeted by HG Q New Claim: PASLoggedOff
13:41:56 76138342mS PRN: SetLoginUser: (this="ES Melissa Reye", b_from_remote=0 ) User NoUser trying to log on to extn 8655, on top of user ES Melissa Reye at UTC time 1463074916
13:41:56 76138342mS CMExtnEvt: LoginUserList: user=<none>(current extn=<none>) goes to extn=8655(current user=ES Melissa Reye)
13:41:56 76138342mS CMExtnEvt: LoginUserList: user=ES Melissa Reye(current extn=<none>) goes to extn=<none>(current user=<none>)
13:41:56 76138343mS CMExtnEvt: NoUser: ExtnFault now 1
13:41:56 76138345mS CMTARGET: c0a8020a00001504 1.1864.1 19712 H323TrunkEP: GetNoAnswerTimer:16
13:41:56 76138345mS CMCallEvt: c0a80a050001b1a8 10315.111016.0 19712 ES Karen Marroq.0: StateChange: END=T CMCSIdle->CMCSOffering
13:41:56 76138345mS CMExtnEvt: ES Karen Marroq: CMExtnHandler::SetCurrent( id: 0->111016 )
13:41:56 76138346mS SIP Tx: TCP 192.168.30.5:5060 -> 10.15.164.10:50245
INVITE sip:8626@10.15.164.10;transport=tcp SIP/2.0
Via: SIP/2.0/TCP 192.168.30.5:5060;rport;branch=z9hG4bK49a9aa6eb109ada74a8782f858c2953d
From: "ATLANTA CTY GOV" <sip:4045467300@192.168.30.5>;tag=a5ce3b67e47d1873
To: <sip:7040@192.168.30.5;transport=tcp>
Call-ID: 65c3da5d08fca8efc854ac4cab538cfd
CSeq: 77138817 INVITE
Contact: "ATLANTA CTY GOV" <sip:4045467300@192.168.30.5:5060;transport=tcp>
Max-Forwards: 70
Allow: INVITE,ACK,CANCEL,OPTIONS,BYE,INFO,REFER,NOTIFY,SUBSCRIBE,REGISTER,PUBLISH,UPDATE
Supported: timer,100rel
User-Agent: IP Office 9.1.6.0 build 153
P-Asserted-Identity: "ATLANTA CTY GOV>Q New Claim" <sip:4045467300@192.168.30.5:5060>
Content-Type: application/sdp
Content-Length: 204

v=0
o=UserA 2009094981 1372913058 IN IP4 192.168.30.5
s=Session SDP
c=IN IP4 192.168.30.5
t=0 0
m=audio 42910 RTP/AVP 0 101
a=rtpmap:0 PCMU/8000
a=rtpmap:101 telephone-event/8000
a=fmtp:101 0-15
13:41:56 76138346mS CMCallEvt: c0a80a050001b173 10348.110963.0 -1 BaseEP: DELETE CMEndpoint d88349d8 TOTAL NOW=466 CALL_LIST=203

********** contact lost with 192.168.10.5 at 13:41:28 12/5/2016 - reselect = 1 **********
******************************************************************

********** SysMonitor v9.1.6.0 build 153 **********

********** contact made with 192.168.10.5 at 13:47:28 12/5/2016 **********

********** System (192.168.10.5) has been up and running for 15secs(15225mS) **********

Isaac Braca
Avaya ACA - IP Office
CTO / ICCS & Co., LLC.


ICCS, Your Premier IT & Telecom Partner, is a New York City Based Avaya SMB Expert Business Partner and IT Consulting Firm.

Visit and Post on my Blog:

http://ipofficeexpert.blogspot.com/

 

[derfloh]

You should get a full trace and not only SIP.

Did you look into the core dump files in WebControl?

 

[superikey]

System crashed again today for the second time at 5PM/
Working with Avaya Support on this as we have IPOSS on the system, and sent them core dumps and monitor traces both time.
Tech indicates we are having a segmentation issue which indicates its a problem in the IPO Code and he is working with R&D to get us a fix.

Is anyone else having this issue?


Isaac Braca
Avaya ACA - IP Office
CTO / ICCS & Co., LLC.


ICCS, Your Premier IT & Telecom Partner, is a New York City Based Avaya SMB Expert Business Partner and IT Consulting Firm.

Visit and Post on my Blog:

http://ipofficeexpert.blogspot.com/

 

[derfloh]

No... We have SP6 virtualized too.

 

[rdoubrava]

We are on 9.1 sp6 virtualized and not having problems. We have only one site and have not rolled out to other locations. Do you have users from site #2 in a HG at site #1 or vise verse?

 

[superikey]

yes absolutely users cross sites for HG membership, users also use trunks from both sites, etc.
note all users are created on the solution and not on the individual switches. extns are on the site specific systems

Isaac Braca
Avaya ACA - IP Office
CTO / ICCS & Co., LLC.


ICCS, Your Premier IT & Telecom Partner, is a New York City Based Avaya SMB Expert Business Partner and IT Consulting Firm.

Visit and Post on my Blog:

http://ipofficeexpert.blogspot.com/

 

[janni78]

Are they logged out if they don't answer HG calls?
Looks like "ES Melissa Reye" is logged out just before it crashes.

"Trying is the first step to failure..." - Homer

 

[superikey]

Yes we have it set to force users to be logged out if they do not answer a HG Call.
ES Melissa Reye is a user in El-Salvador Remote Call Center
She used One-X + Communicator on a Windows PC along with a USB Headset.

Isaac Braca
Avaya ACA - IP Office
CTO / ICCS & Co., LLC.


ICCS, Your Premier IT & Telecom Partner, is a New York City Based Avaya SMB Expert Business Partner and IT Consulting Firm.

Visit and Post on my Blog:

http://ipofficeexpert.blogspot.com/

 

[superikey]

Further, after spending 3 hours on the phone with Avaya today, they confirmed there is a critical patch for SE 9.1.6 Systems, where when a HG user either DND's to ignore a call or is logged out when they don't answer, the system reboots. For now we removed DND buttons from all users, and set all HG's Agent status on no Answer... setting to None while we wait for T4 to further review our logs and see if this in fact is our issue.

Any one else having this?
Janni78 where you the one who had this problem and worked with Gordon @ Avaya?

Isaac Braca
Avaya ACA - IP Office
CTO / ICCS & Co., LLC.


ICCS, Your Premier IT & Telecom Partner, is a New York City Based Avaya SMB Expert Business Partner and IT Consulting Firm.

Visit and Post on my Blog:

http://ipofficeexpert.blogspot.com/

 

[rdoubrava]

So in Users - Telephony - Supervisor settings
You are using the Status on No-Answer, set for Log-off.
Have you tried another setting?
I don't use this feature. It was broken in several versions of 8.0 and 8.1
Symptoms were system reboots, hunt groups issues, etc.

 

[janni78]

No, I just looked at the log on what happened before the crash.
Most installs don't use that feature and it can be a combination of how your system is setup so most will never experience the issue.

"Trying is the first step to failure..." - Homer

 

[superikey]

Update
Though system not as busy on the weekend, After removing DND buttons from all phones, and turning off agent status on no answer for HG's, system has thus far been stable.

Isaac Braca
Avaya ACA - IP Office
CTO / ICCS & Co., LLC.


ICCS, Your Premier IT & Telecom Partner, is a New York City Based Avaya SMB Expert Business Partner and IT Consulting Firm.

Visit and Post on my Blog:

http://ipofficeexpert.blogspot.com/

 

[joe2938]

If this is really the issue then I would open up a case with Avaya support.