Strange Problem after upgrade to 8.1(63) 1

[PhoneGuyNJ]

I have IP Office 500V2, with a Digital and a Combo Card. We have 1 POTS line on port 9 of the combo, and all was working fine until the upgrade. Now, almost every 30 minutes, the system thinks that line is ringing, when in fact it's not. It rings about 1 ring and stops, with no callerID. I have now set an incoming call route with a "?" to go to a phanthom extension that just is a busy, because it's annoying. Tried everything, reboots, some settings, etc. Strange because it's only since the upgrade. It seems as though it is detecting "something" but I dont know what that something is. I can say there is no voicemail on the POTS.

Here is a piece of the log when it happens if anyone has any ideas..

2013-01-10T13:58:48 4647038mS ATMChannel: [1] DTMF Message Rx: 0x0000 (Tone=0x0000)
2013-01-10T13:58:48 4647038mS ATMChannel: [1] CLI Message Rx:
2013-01-10T13:58:48 4647038mS ATMChannel: end of CLI Message
2013-01-10T13:58:48 4647038mS ATMChannel: [1] StateChange Idle->CLIAwaitData
2013-01-10T13:58:48 4647038mS ATMChannel: [1] CMLinkLayer Tx: 'Alert_PreIndication'
2013-01-10T13:58:48 4647038mS ATMIO: [1] CLI DETECTION OFF, EQUALISER OFF
2013-01-10T13:58:48 4647038mS ATMChannel: [1] StateChange CLIAwaitData->CLIDataSettle
2013-01-10T13:58:48 4647039mS ATMCMLine: [1] CMLinkLayer Rx: 'Alert_PreIndication' callid=(lid=1, id=2, in=1)
2013-01-10T13:58:48 4647039mS ATMCMLine: [1] StateChange Idle->IncomingGuard
2013-01-10T13:58:48 4647238mS ATMChannel: [1] Sloppy timeout(200ms)
2013-01-10T13:58:48 4647238mS ATMChannel: [1] StateChange CLIDataSettle->Incoming
2013-01-10T13:58:48 4647238mS ATMChannel: [1] CMLinkLayer Tx: 'Ringing'
2013-01-10T13:58:48 4647238mS ATMCMLine: [1] CMLinkLayer Rx: 'Ringing' callid=(lid=1, id=2, in=1)
2013-01-10T13:58:48 4647239mS ATMCMLine: [1] StateChange IncomingGuard->Present
2013-01-10T13:58:48 4647239mS ATMCMLine: [1] IncomingCall
2013-01-10T13:58:48 4647240mS CMCallEvt: 0.1113.0 -1 BaseEP: NEW CMEndpoint f5170318 TOTAL NOW=1 CALL_LIST=0
2013-01-10T13:58:48 4647240mS CMCallEvt: CREATE CALL:6 (f51889b4)
2013-01-10T13:58:48 4647240mS CMCallEvt: 0.1114.0 -1 BaseEP: NEW CMEndpoint f51874e8 TOTAL NOW=2 CALL_LIST=0
2013-01-10T13:58:48 4647242mS CMLineRx: v=1
CMSetup
Line: type=AnalogueLine 1 Call: lid=1 id=2 in=1
Called[] Type=Unknown (0) Reason=CMDRdirect SndComp Calling[] Type=Unknown Plan=Unknown Pres=NAInterworking (2)
BC: CMTC=Speech CMTM=Circuit CMTR=64 CMST=Default CMU1=ULaw
BChan: slot=1 chan=1
IE CMIEDeviceDetail (231) LOCALE=enu HW=15 VER=8 class=CMDeviceAlogTrunk type=0 number=1 channel=0 rx_gain=32 tx_gain=32 ep_callid=2 ipaddr=192.168.42.1 apps=0
2013-01-10T13:58:48 4647242mS CD: CALL: 1.2.1 BState=Idle Cut=1 Music=0.0 Aend="Line 1" (1.1) Bend="" [] (0.0) CalledNum= () CallingNum= () Internal=0 Time=2 AState=Idle
2013-01-10T13:58:48 4647243mS CMCallEvt: 1.2.1 6 Alog Trunk:1: StateChange: END=A CMCSIdle->CMCSDialInitiated
2013-01-10T13:58:48 4647243mS CMTARGET: 1.2.1 6 Alog Trunk:1: LOOKUP CALL ROUTE: type=0 called_party= sub= calling= dir=in complete=1 ses=0
2013-01-10T13:58:48 4647243mS CMTARGET: 1.2.1 6 Alog Trunk:1: SET BESTMATCH: length 0 vs -1 match= dest=Home Line
2013-01-10T13:58:48 4647243mS CMTARGET: 1.2.1 6 Alog Trunk:1: AMBIGUITY: match= dest=498
2013-01-10T13:58:48 4647243mS CMTARGET: 1.2.1 6 Alog Trunk:1: SET BESTMATCH: length 0 vs 0 match= dest=498
2013-01-10T13:58:48 4647243mS CMCallEvt: Priority hike: call 6 priority 0->1
2013-01-10T13:58:48 4647243mS CMTARGET: 1.2.1 6 Alog Trunk:1: LOOKUP ICR: DDI=856-854-2679 CGPN= (Destination 498 ) => CDPN=498
2013-01-10T13:58:48 4647244mS CMTARGET: 1.2.1 6 Alog Trunk:1: ADD TARGET (N): number=498 type=0 depth=1 nobar=1 setorig=1 ses=0
2013-01-10T13:58:48 4647244mS CMTARGET: 1.2.1 6 Alog Trunk:1: SET USER: BUSY orig=1
2013-01-10T13:58:48 4647244mS CMTARGET: 1.2.1 6 Alog Trunk:1: ADD USER: BUSY depth=1 disallow_cw=0 dnd=0 real_call=1 group_call=0 type(CMNTypeUnknown) incl(0x0) excpt(0x0), allow_redir(1) remote=00000000 simult 0 (0)
2013-01-10T13:58:48 4647244mS CMTARGET: TryLoggedOutTargeting for user BUSY Failed
2013-01-10T13:58:48 4647244mS CMTARGET: 1.2.1 6 Alog Trunk:1: NO INITIAL TARGETS: BUSY
2013-01-10T13:58:48 4647244mS CMTARGET: 1.2.1 6 Alog Trunk:1: ADD USER: BUSY depth=1 disallow_cw=0 dnd=0 real_call=1 group_call=0 type(CMNTypeUnknown) incl(0x40) excpt(0x40), allow_redir(1) remote=00000000 simult 0 (0)
2013-01-10T13:58:48 4647245mS CMTARGET: TryLoggedOutTargeting for user BUSY Failed
2013-01-10T13:58:48 4647245mS CMTARGET: 1.2.1 6 Alog Trunk:1: SELECT: TRY VOICEMAIL orig_hg() orig_user(498)
2013-01-10T13:58:48 4647245mS CMTARGET: 1.2.1 6 Alog Trunk:1: Fallback() targeting failed
2013-01-10T13:58:48 4647246mS CMLOGGING: CALL:2013/01/1013:58,00:00:00,000,,I,498,,,,,0,,""n/a,0
2013-01-10T13:58:48 4647246mS CD: CALL: 1.2.1 BState=Idle Cut=0 Music=0.0 Aend="Line 1" (1.1) Bend="BUSY(498)" [] (0.0) CalledNum=498 (BUSY) CallingNum= () Internal=0 Time=6 AState=Dialling
2013-01-10T13:58:48 4647246mS CD: CALL: 1.2.1 Deleted
2013-01-10T13:58:48 4647246mS CMTARGET: 1.2.1 -1 Alog Trunk:1: ~CMTargetHandler f518a7e0 ep f5170318
2013-01-10T13:58:48 4647246mS CMCallEvt: 1.2.1 -1 Alog Trunk:1: StateChange: END=X CMCSDialInitiated->CMCSCompleted
2013-01-10T13:58:48 4647247mS CMCallEvt: 0.1114.0 -1 BaseEP: DELETE CMEndpoint f51874e8 TOTAL NOW=1 CALL_LIST=0
2013-01-10T13:58:48 4647247mS CMCallEvt: END CALL:6 (f51889b4)
2013-01-10T13:58:48 4647247mS ATMCMLine: [1] ControlEchoCancellation ON
2013-01-10T13:58:48 4647247mS ATMCMLine: [1] CMLinkLayer Tx: 'EchoControl'
2013-01-10T13:58:48 4647249mS ATMChannel: [1] CMLinkLayer Rx: 'EchoControl' (ls)
2013-01-10T13:58:48 4647249mS ATMIO: [1] ECHO CANCELLATION ON
2013-01-10T13:58:48 4647249mS PRN: Config Write Wake Up
2013-01-10T13:58:48 4647513mS RES: Thu 10/1/2013 13:58:49 FreeMem=60075060(1) CMMsg=5 (6) Buff=5200 956 999 12463 5 Links=7775 BTree=0 CPU=10/13/16495/34376/36055/2
2013-01-10T13:58:48 4647513mS RES2: IP 500 V2 8.1(63) Tasks=48 RTEngine=0 CMRTEngine=0 ExRTEngine=0 Timer=58 Poll=0 Ready=0 CMReady=0 CMQueue=0 VPNNQueue=0 Monitor=2 SSA=1 TCP=21 TAPI=1 ASC=1 SYS=MNTD OPT=UMNT SDSPD=2034
2013-01-10T13:58:48 4647513mS RES4: XML MemObjs=45 PoolMem=2097152(1) FreeMem=2082064(1)
2013-01-10T13:58:48 4647554mS PRN: Updates IO list size 1 updated list size 1
2013-01-10T13:58:48 4647554mS PRN: Sending Updates out to f5b05870 started
2013-01-10T13:58:48 4647554mS PRN: Sending Updates out to f5b05870 finished
2013-01-10T13:58:48 4647555mS PRN: Config Write Completed
2013-01-10T13:58:53 4652238mS ATMChannel: [1] Sloppy timeout(5000ms)
2013-01-10T13:58:53 4652238mS ATMIO: [1] TRUNK RELEASE
2013-01-10T13:58:53 4652238mS ATMChannel: [1] StateChange Incoming->Idle
2013-01-10T13:58:53 4652238mS ATMIO: [1] ECHO CANCELLATION OFF
2013-01-10T13:58:53 4652238mS ATMIO: [1] BLOCK FORWARDING OFF
2013-01-10T13:58:53 4652238mS ATMIO: [1] CLI DETECTION ON, EQUALISER OFF
2013-01-10T13:58:53 4652238mS ATMChannel: [1] CMLinkLayer Tx: 'Disconnect'
2013-01-10T13:58:53 4652239mS ATMCMLine: [1] CMLinkLayer Rx: 'Disconnect' callid=(lid=1, id=0, in=1)
2013-01-10T13:58:53 4652239mS ATMCMLine: [1] StateChange Present->DiscInd
2013-01-10T13:58:53 4652239mS ATMCMLine: [1] CM Tx: 'CMReleaseComp' callid=(lid=1, id=2, in=1)
2013-01-10T13:58:53 4652239mS CMLineRx: v=1
CMReleaseComp
Line: type=AnalogueLine 1 Call: lid=1 id=2 in=1
BChan: slot=1 chan=1
IE CMIERespondingPartyNumber (230)(P:2 S:100 T:0 N:0 R:4) number=
IE CMIEDeviceDetail (231) LOCALE=enu HW=15 VER=8 class=CMDeviceAlogTrunk type=0 number=1 channel=0 rx_gain=32 tx_gain=32 ep_callid=2 ipaddr=192.168.42.1 apps=0
2013-01-10T13:58:53 4652239mS ATMCMLine: [1] StateChange DiscInd->Idle
2013-01-10T13:58:53 4652240mS ATMCMLine: [1] CMLinkLayer Tx: 'DisconnectAck'
2013-01-10T13:58:53 4652240mS CMCallEvt: 1.2.1 -1 Alog Trunk:1: StateChange: END=X CMCSCompleted->CMCSDelete
2013-01-10T13:58:53 4652240mS CMCallEvt: 1.2.1 -1 BaseEP: DELETE CMEndpoint f5170318 TOTAL NOW=0 CALL_LIST=0
2013-01-10T13:58:53 4652241mS ATMChannel: [1] CMLinkLayer Rx: 'DisconnectAck' (ls)
2013-01-10T13:58:53 4652513mS RES: Thu 10/1/2013 13:58:54 FreeMem=60081100(1) CMMsg=5 (6) Buff=5200 956 999 12463 5 Links=7778 BTree=0 CPU=3/4/16495/34711/36055/2
2013-01-10T13:58:53 4652513mS RES2: IP 500 V2 8.1(63) Tasks=48 RTEngine=0 CMRTEngine=0 ExRTEngine=0 Timer=57 Poll=0 Ready=0 CMReady=0 CMQueue=0 VPNNQueue=0 Monitor=2 SSA=1 TCP=21 TAPI=1 ASC=1 SYS=MNTD OPT=UMNT SDSPD=2034
2013-01-10T13:58:53 4652513mS RES4: XML MemObjs=45 PoolMem=2097152(1) FreeMem=2082064(1)

 

[yankblan]

We're trying to find regular copper to run the same tests, but those are getting rare...

http://interconnexionsld.ca/

 

[SIRMcD]

Ok , I have this issue resolved but still requires more data to present to Avaya. Thanks to yankblan for pointing me in the right direction. This is a MWI burst of information. The fibre fed ONT device has a back end(at least with Bell) of Broadsoft as opposed to dms-100. I spoke to our techs in that department and they had run into a similar situation in Quebec. He was able to turn off the MWI indicator. Note that the customer did NOT have voicemail active on the line. The MWI is separate and left on whether or not they have have voicemail. After having it disabled I remained on site and no phantom rings for over an hour. He confirmed that this is standard operation of the Fibre to the business line. The burst occurs every half hour.
Going forward the people in that department will be testing with Our Avaya support group to see what can be done in future releases to stop the Avaya from ringing. Obviously there is an issue here with the Avaya as it only rings on certain releases. If I hear any more info I'll post here.
I would not be surprised if Verizon also uses the broadsoft platform as it also the back-end used in Hosted IP Voice applications.

 

[amriddle01]

Thanks for the update, being UK based we don't and will not see this issue, but I have archived this post so I can direct other folks to it, this is bound to rear it's ugly head again. I suspect they change the sensitivity of the line card from release to release as some folks complain of it not detecting ringing on long run/weak lines etc, so it's a balancing act for Avaya



"No problem monkey socks

 

[PhoneGuyNJ]

So did you have the provider turn this off? Looks like this is the exact same issue with Verizon.

 

[SIRMcD]

Yes, I had the provider turn off MWI. In my case this is fairly easy as the company I work for is a major carrier. So it's just matter of calling an internal department. I can tell you this , it was a real hunt to find the correct department. Because the fibre to the business is so new to the company , few people have seen any issue's. I think if you contacted Verizon and ask them to turn it off you may have some luck.
I can imagine it would be extremely difficult to make a product that has to connect to such a wide range of carriers all over the world.
I think that was why Nortel was so rock solid on some things, they mostly designed the equipment for the canadian market and DMS100.

 

[yankblan]

They should add in the trunk settings a detection level setting,or like Panasonic, let a delayed ring (3 to 5 seconds) on caller ID settings. All we have is either clid on or off, which is not good. And yes I am in Québec, and Bell and Vidéotron have been inundated with this issue with Avaya customer since release 8.0 . Fortunately we are major partners with Vidéotron and have provided us stuff to lab with.

To quote South Park, blame Canada!

http://interconnexionsld.ca/

 

[tlpeter]

I suspect this "new" 7.0 feature is the cause that this now happens:

3.11 Optimization for CallerID Detection on Analogue trunks
In IP Office Release 7.0 the timings for Analogue Trunks in relation to CallerID detection have been optimized. There should be no noticeable changes except for faster responses to incoming calls, particularly where the trunk is configured for incoming callerID but none is presented.

BAZINGA!

I'm not insane, my mother had me tested!

 

[PhoneGuyNJ]

I saw someplace a trick to actually add an announcement to the incoming group with no recording, and then set it to ringing - which will actually pick the line up (it's already doing that anyhow)before it routyes to the hunt group members. I wonder if this trick will work...

 

[PhoneGuyNJ]

I see a new Service Pack 5 is out - before I go perform this upgrade, I was just wondering if anyone has already done that - and has any insight on if this problem seems to be fixed.

I don't see anything related in the release notes, but that doesn't always mean it's not a symptom of something else.

 

[yankblan]

We were told by Avaya that this problem will not be addressed in release 8.1, but will be corrected for 9.1 (apparently they will skip 9.0; apparently again). So stick to 8.56 if you don't have SCN and VM Pro.

Anyway, if you have SCN and VMPro, you should be using PRI or SIP...

http://interconnexionsld.ca/

 

[PhoneGuyNJ]

That does not make a whole lot of sense. Besides downgrading or leaving at an older revision - since it's been quite some time, how has others got around this? Verizon seems clueless on the issue when opening service requests with them. Has anyone tried any thing else at all to stop the initial ring from happening? Usually I can come up with more than one way to skin a cat in this kind of issue, but I am at a loss on this one.

 

[yankblan]

We basically went with older releases; thing I didn't try are the new combo and ATM4U v2 cards; don't know if they handle that problem better.

http://interconnexionsld.ca/

 

[PhoneGuyNJ]

Well there are others that had this issue, and probably got around it somehow. I am going to call Verizon one more time to see if they can do anything.. Surely someone has fixed this on their own. I am curious as to where they are stating 9.1 on this issue - seems silly.

 

[mikedphones]

My 2 cents

just took ours to 8.1(69) from (43) and now I can set my watch to a phantom ring every 30 minutes. We have a mix of lines, Verizon copper, SIP, Comcast analog handoff, all are quiet, the only line that rings is the FiOS every 30 minutes. We use Combo and DS 8 mods with ATM V1 daughter cards. I am going back to (43) although I think (57) as stated is ok for this too. There is the Park button issue on (67) so that needs to be avoided in many cases too.

What's the lowest build that we can use ATMV2 mods on? Is this going to be an issue with FiOS if we can't downgrade low enough with those mods in the game?

 

[PhoneGuyNJ]

@mikedphones - that is exactly the problem. The IP Office is picking up a refresh signal every 30 minutes. I am not about to spend any more money on this to buy ATM V2 if there is no proof this fixes the issue.
I did spend close to an hour on the phone with Verizon, and they told me I was crazy since we had VM turned off, and all they kept wanting to do was reboot the ONT, which doesn't do any good at all. It was like pulling teeth.

Another interesting thing to note, is even though we downgraded, in System Status application, it still flags those calls as incoming abandoned.. just does not ring through like it does in later releases. AS WELL, we went on a service call for an older 4.2 system, and for the heck of it, looked at system monitor (they have FIOS as well) - and it ALSO flags those abandons! (Yes, I know, we have been trying to get them to upgrade for quite a while now).

For the time being, we went into Verizons online settings and setup call forwarding on the POTS to a spare DID coming through SIP - and that solved it so that we can bring them to the latest release at this point.

Again, since FIOS is fairly common, I am amazed that either side of the house can't seem to understand this.

 

[mikedphones]

Thanks Phone Guy. I went back to 8.1(43) all is well for now, however, I will have to do a dry run on a high enough build supporting an ATMV2 for a site with Verizon Fios to see if I can even install at all. Otherwise, I will have to be sure to have an ATMv1 on hand.

 

[SIRMcD]

Verizon needs to turn off MWI or VMWI, ask them to turn off both just to see if it fixes the issue(it will). Official stance for most carriers will be to downgrade the IPO to a software that does not cause this until Avaya officially fixes the problem.
It is not an actual problem with the FIOS service. This is normal operation. It just updates MWI more often than standard switching equipment. That update to MWI is a 20 volt AC burst.