older mitel 3300 with digital phones and PER nodes alarms

[Matty269]

system is running ver 6 mcd and still using digital phones having connection issues to PER nodes right after the system syncs to the license AMC happens 2-3 times a week all devices in the PER nodes go offline and come back online

from the error logs cut and pasted from an excel file it starts at the bottom at 12:55 pm right after it synced to AMC then looses connection to the PER nodes

any thoughts or insight would be great


SecurityMgr RequestMatchingProfileData() System is running with GDM Mode. Attempting to retrieve user profile.
Call Control - Software Program: dvcmgr Offset: 2:000032A4 PC: 0029EF72 Program: dvcmgr Offset: 2:00004542 PC: 002A0210
Call Control - Software CBQUEUE: Swid $02, $90 not found in callback queue.Unable to mature callback set against DN 5999.Traceback: Program: cbqueue Offset: 2:0000128A PC: 0038E088 Program: dvcmgr Offset: 2:00002C40 PC: 0029E90E
Call Control - Maintenance MC330AB DNI Line at location 07 1 06 installed
Call Control - Maintenance MC330AB DNI Line at location 07 1 05 installed
Call Control - Maintenance MC330AB DNI Line at location 07 1 02 installed
Call Control - Maintenance MC330AB DNI Line at location 07 1 01 installed
Call Control - Maintenance MC330AB DNI Line at location 07 1 07 installed
Call Control - Maintenance MC330AB DNI Line at location 07 1 10 installed
Call Control - Maintenance MC330AA DNI Line at location 05 1 04 installed
Call Control - Maintenance MC330AB DNI Line at location 05 1 03 installed
Call Control - Maintenance MC330AA DNI Line at location 05 1 02 installed
Call Control - Maintenance MC330AB DNI Line at location 07 1 03 installed
Call Control - Maintenance MC330AB DNI Line at location 05 1 01 installed
Call Control - Maintenance MC330AB DNI Line at location 07 1 12 installed
Call Control - Maintenance MC330AB DNI Line at location 07 1 11 installed
Call Control - Maintenance MC330AB DNI Line at location 07 1 08 installed
Call Control - Software dnichdlr - invalid plid : 7 1 3 9 1TRACEBACK (see above for first level info): dnichdlr +2:$DE6 dnichdlr +2:$110A dnichdlr +2:$1164
Call Control - Software INVALID MESSAGE, OFFSET 2:$2E6, PC: $2E629ALOG_ID: dnichdlr#20, PROCEDURE: valid_dnic_msg, SEVERITY: cp_informPID: ($4, $6C)MSG_FUNC_CODE: $F0, MSG_TX_PID: ($2, $10)MSG_DATA: $0 $0 $0 $7 $0 $33 $33 $30 $41 $42 $20 $6 $20
Call Control - Maintenance MC330Ax DNI Line at location 07 1 05 removed
Call Control - Maintenance MC330Ax DNI Line at location 05 1 04 removed
Call Control - Maintenance MC330Ax DNI Line at location 07 1 01 removed
Call Control - Maintenance MC330Ax DNI Line at location 05 1 01 removed
Call Control - Maintenance MC330Ax DNI Line at location 05 1 03 removed
Call Control - Maintenance MC330Ax DNI Line at location 05 1 02 removed
Call Control - Maintenance MC330Ax DNI Line at location 07 1 02 removed
Call Control - Maintenance MC330Ax DNI Line at location 07 1 07 removed
Call Control - Maintenance MC330Ax DNI Line at location 07 1 10 removed
Call Control - Maintenance MC330Ax DNI Line at location 07 1 06 removed
Call Control - Maintenance MC330Ax DNI Line at location 07 1 03 removed
Call Control - Maintenance MC330Ax DNI Line at location 07 1 12 removed
Call Control - Maintenance MC330Ax DNI Line at location 07 1 11 removed
Call Control - Maintenance MC330Ax DNI Line at location 07 1 08 removed
Call Control - Maintenance MC312AB Peripheral Cont I at location 5 1 16 DTMF Receiver circuit 7 was returned to service
Call Control - Maintenance MC312AB Peripheral Cont I at location 7 1 16 DTMF Receiver circuit 7 was returned to service
Call Control - Software DAMAUDITS: no card at internal plid 6 0 10 28 Dam state = $8
Call Control - Software DAMAUDITS: no card at internal plid 6 0 10 6 Dam state = $8
Call Control - Software DAMAUDITS: no card at internal plid 6 0 9 12 Dam state = $8
Call Control - Maintenance MC312AB Peripheral Cont I at location 7 1 16 DTMF Receiver circuit 1 was returned to service
Call Control - Maintenance MC312AB Peripheral Cont I at location 5 1 16 DTMF Receiver circuit 1 was returned to service
OPS Alarms DSA ForwardUDTMessageToOPS()->Cannot connect to UDTHANDLER
OPS Alarms DSA ForwardUDTMessageToOPS()->Cannot connect to UDTHANDLER
Call Control - Software OS (DDAUDIT) ==> Controller number $7, plane 0 does not respond to audit
Call Control - Software OS (Auditwork) error: Audit assert failed for audit Data Distributio.
Call Control - Maintenance ------------ Current System Alarm : | NO ALARM | viewed from Active ------------
Call Control - Maintenance The link from MCECAAA MN3300 Controller at location 01 1 01 Active to MC312AB Peripheral Cont I at location 07 1 16 Activeis now OPEN.
Call Control - Maintenance MC312AB Peripheral Cont I at location 07 1 16 installed
Call Control - Maintenance The link from MCECAAA MN3300 Controller at location 01 1 01 Active to MC312AB Peripheral Cont I at location 05 1 16 Activeis now OPEN.
Call Control - Maintenance MC312AB Peripheral Cont I at location 05 1 16 installed
FimAL MsgLink::HandleUapiCloseLink() -> MSS registered Msg Link 33 for MMCNum 3, FIMNum 0 MsgLinkNum 0 is closed for cause 7.
Call Control - Software MSGmgr : Message link from plane 0 to plane 0 for controller 7is scanning due to cause value 7
Call Control - Software MSGmgr : Message link from plane 0 to plane 0 for controller 7is scanning due to cause value 0
Call Control - Software MSGmgr : Message link from plane 0 to plane 0 for controller 7is scanning due to cause value 0
MSGLINKMGR Could not detect the presence of the FIM connected unit for at least 1 second in link 12. ICB = 0x6
MSGLINKMGR ActionRoutine07 --> Link Closed in link 12 cause = 7 (lower node not available in open state) ICB = 0x6 .
Call Control - Maintenance ---------------+--------+-------+-------+-------+---------+---------+---------- Per msg link | 4 | 2 | 50 % | Minor | 50 % | 80 % | 100 % | ---------------+--------+-------+-------+-------+---------+---------+----------
Call Control - Maintenance ---------------+--------+-------+-------+-------+---------+---------+---------- |Total In| Unavailable | Alarm Threshold Percentages Category | System | Total | % | Alarm | MINOR | MAJOR | CRITICAL
Call Control - Maintenance ------------ Current System Alarm : | MINOR | viewed from Active ------------
FimAL MsgLink::HandleUapiCloseLink() -> MSS registered Msg Link 21 for MMCNum 1, FIMNum 1 MsgLinkNum 0 is closed for cause 7.
Call Control - Software MSGmgr : Message link from plane 0 to plane 0 for controller 5is scanning due to cause value 7
Call Control - Software MSGmgr : Message link from plane 0 to plane 0 for controller 5is scanning due to cause value 0
Call Control - Software MSGmgr : Message link from plane 0 to plane 0 for controller 5is scanning due to cause value 0
MSGLINKMGR Could not detect the presence of the FIM connected unit for at least 1 second in link 6. ICB = 0x6
MSGLINKMGR ActionRoutine07 --> Link Closed in link 6 cause = 7 (lower node not available in open state) ICB = 0x6 .
Call Control - Maintenance The link from MCECAAA MN3300 Controller at location 01 1 01 Active to MC312Ax Peripheral Cont I at location 07 1 16 is now in SCANNING mode.
Call Control - Maintenance MC312Ax Peripheral Cont I at location 07 1 16 removed
FimAL MsgLink::HandleUapiCloseLink() -> MSS registered Msg Link 33 for MMCNum 3, FIMNum 0 MsgLinkNum 0 is closed for cause 6.
Call Control - Software MSGmgr : Message link from plane 0 to plane 0 for controller 7is scanning due to cause value 6
Call Control - Software MSGmgr : Message link from plane 0 to plane 0 for controller 7is scanning due to cause value 0
MSGLINKMGR ActionRoutine06 --> Link Closed in link 12 cause = 6 (4 = checksum, 5 = parity, 6 = expired from waiting for the lower node ) in open state ICB = 0x6 .
FimAL MsgLink::HandleUapiCloseLink() -> MSS registered Msg Link 21 for MMCNum 1, FIMNum 1 MsgLinkNum 0 is closed for cause 7.
Call Control - Software MSGmgr : Message link from plane 0 to plane 0 for controller 5is scanning due to cause value 7
Call Control - Software MSGmgr : Message link from plane 0 to plane 0 for controller 5is scanning due to cause value 0
MSGLINKMGR Could not detect the presence of the FIM connected unit for at least 1 second in link 6. ICB = 0x6
MSGLINKMGR ActionRoutine07 --> Link Closed in link 6 cause = 7 (lower node not available in open state) ICB = 0x6 .
Call Control - Maintenance The link from MCECAAA MN3300 Controller at location 01 1 01 Active to MC312AB Peripheral Cont I at location 07 1 16 Activeis now OPEN.
Call Control - Maintenance MC312AB Peripheral Cont I at location 07 1 16 installed
Call Control - Maintenance The link from MCECAAA MN3300 Controller at location 01 1 01 Active to MC312Ax Peripheral Cont I at location 07 1 16 is now in SCANNING mode.
Call Control - Maintenance The link from MCECAAA MN3300 Controller at location 01 1 01 Active to MC312Ax Peripheral Cont I at location 05 1 16 is now in SCANNING mode.
Call Control - Maintenance MC312Ax Peripheral Cont I at location 07 1 16 removed
Call Control - Maintenance MC312Ax Peripheral Cont I at location 05 1 16 removed
FimAL MsgLink::HandleUapiCloseLink() -> MSS registered Msg Link 33 for MMCNum 3, FIMNum 0 MsgLinkNum 0 is closed for cause 7.
Call Control - Software MSGmgr : Message link from plane 0 to plane 0 for controller 7is scanning due to cause value 7
Call Control - Software MSGmgr : Message link from plane 0 to plane 0 for controller 7is scanning due to cause value 0
FimAL MsgLink::HandleUapiCloseLink() -> MSS registered Msg Link 21 for MMCNum 1, FIMNum 1 MsgLinkNum 0 is closed for cause 7.
Call Control - Software MSGmgr : Message link from plane 0 to plane 0 for controller 5is scanning due to cause value 7
Call Control - Software MSGmgr : Message link from plane 0 to plane 0 for controller 5is scanning due to cause value 0
MSGLINKMGR Could not detect the presence of the FIM connected unit for at least 1 second in link 12. ICB = 0x6
MSGLINKMGR ActionRoutine07 --> Link Closed in link 12 cause = 7 (lower node not available in open state) ICB = 0x6 .
MSGLINKMGR Could not detect the presence of the FIM connected unit for at least 1 second in link 6. ICB = 0x6
MSGLINKMGR ActionRoutine07 --> Link Closed in link 6 cause = 7 (lower node not available in open state) ICB = 0x6 .
Licensing AMC sync 3300-12.0.9=software

 

[Lundah]

Are both peripheral cabinets connected to the same FIM card in the controller? If so, try replacing that. There may be a link between the license sync as well. Might need to run a backup, completely re-load the system software from scratch, re-license, then restore the backup.

 

[nytalkin]

How many DNI cards in the system. If greater than 20 use Lundah's solution. Still could be a bad FIM though.

I suppose you're entitled to your opinion, I'm just not going to suppose very hard.

 

[Matty269]

update we had issues with a network scanner early last year we told the customer IT team to stop scanning any Mitel devices the scans were causing the system to lock up and need to be rebooted. they did and the system calmed down except for the above alarms twice a week or so this holiday season we were scheduled to do some work on the pbx with IP addresses the customers IT dept said from dec 14 th to Jan 4 th there will be no work done on any network devices this must mean no scans either cause the alarms stopped on dec 14th and they came back on jan 4th like clock work of course the IT team won't lend any info

 

[SXWizard]

I have seen issues with network scanners trying to do FTP connections to the controller and causing the lock up as you describe. The IT department needs to delist the controller IP from scanning

 

[EYES]

Matty269,
How did you resolve this? I have a 3300 MXeIII and I am having the exact same issue with a per node, I replaced the power supply board thinking it was failing but that wasn't it. I'm now considering a FIM card but that seems unlikely since it happens right after AMC snyc. The IT department is not scanning this so that can't be it. Any suggestions would be greatly appreciated. -Thanks

 

[Matty269]

EYES,
I have not resolved this as of yet but I am convinced it is some security device on their network causing this. my reasoning:
1. the alarms are like clock work right after an AMC sync in both controllers at this site when they stopped all network work around the holidays the alarms stopped in both controllers during the time period the IT dept set
2. Have you looked for FTP Daemon entries in your logs? I have found entries in both controllers for an IP address from their offsite data center trying to gain ftp access again no entries during the holiday schedule
3. two controllers having the exact same issue? three with yours now?
4. I spoke to the AMC to see if they saw any issues with these two controllers syncing with the AMC and they said it all looked normal
5. I asked if they had some kind of device scanning for activity on port 443 that activates when outgoing traffic is seen and then it gets involved some how? and they said no the IT dept is not real willing to share info when I ask why their offsite data center is trying to talk to the mitel after they said they are not scanning they won't respond
6. I also thought maybe it could be the new root cert needed since 2021 but I keep coming back to why did the issue stop during the holiday schedule? a hardware failure is not going to clear it self if it is bad it is bad
I feel it could also be some kind of auto update that was done for security on port 443 by one of the major firewall mfgr's but again I asked what kind firewall they use and never got a response from the IT team like it's none a my business , at this point I have sent the customer the notice from the Mitel knowledge base that scanning can cause issues and that we see traffic from an offsite device trying to gain access to the mitel If I do resolve this I will pass along any updates good luck I hope this helps

 

[EYES]

Matty269,
I haven't noticed but will look for any Daemon entries, you might be on to something with the firewall scanning port 443. I monitored the controller on site this morning to see what it did at 7:25 AM that's when it typically contacts AMC and subsequently loses the connection to the cabinet. I had the cover off of the Node to see if any LEDs went out on the FIM card(s) but no hiccups this morning. It has gone without a hitch since 2/3/23 which was the day after I swapped out the power supply board. It very briefly went out Friday (3 min) but quickly reconnected and has been normal for the last three mornings so far, but I doubt it will stay that way. I did have a day when the controller lost the cabinet connection for 50 minutes so that is when I thought it might have a failing power board. We are in the process of replacing all DNI phones with IP versions but still have a good old Nupoint messenger voicemail server connected to the cabinet, it has quite of few call boxes configured doing DID call routing so there are potentially serious ramifications when the Node is offline. You've given me some other things to look into much appreciated! I will update you with any positive results. - Thanks Again

 

[Matty269]

Eyes
the connection loss in my case is momentary the cabinet comes right back in like a minute every time but when they were doing an intrusive scan both controllers lost their mind they needed to be reset by the front panel that hasn't happened again this site is a large health care campus which was taken over by a regional medical group so they are now part of a large data network structure where you have a group departments handling a specific area of the network a firewall group and security group a switch group and so on, so it is tough some times to get the answers needed I will keep posting any updates

 

[EYES]

Matty269,

Update:

I was hoping this issue was resolved since all went smooth for 10 days straight, but the system lost connection again this morning. After looking through log categories the error problem happens at the exact same time as an error detecting the FIM card. This is leading me to suspect the FIM card as possibly the source of the problem.


Under Maintenance Warning Logs
1571 2023/Feb/13 07:28:24 Call Control - Maintenance ------------ Current System Alarm : | NO ALARM | viewed from Active ------------
1568 2023/Feb/13 07:27:34 Call Control - Maintenance ---------------+--------+-------+-------+-------+---------+---------+---------- Lines | 602 | 77 | 12 % | Minor | 10 % | 20 % | 100 % | ---------------+--------+-------+-------+-------+---------+---------+----------
1567 2023/Feb/13 07:27:34 Call Control - Maintenance ---------------+--------+-------+-------+-------+---------+---------+---------- |Total In| Unavailable | Alarm Threshold Percentages Category | System | Total | % | Alarm | MINOR | MAJOR | CRITICAL
1566 2023/Feb/13 07:27:34 Call Control - Maintenance ------------ Current System Alarm : | MINOR | viewed from Active ------------
1562 2023/Feb/13 07:27:12 Call Control - Maintenance MC330Ax DNI Line at location 06 1 08 removed
1561 2023/Feb/13 07:27:11 Call Control - Maintenance MC330Ax DNI Line at location 06 1 07 removed
1560 2023/Feb/13 07:27:10 Call Control - Maintenance MC330Ax DNI Line at location 06 1 06 removed
1557 2023/Feb/13 07:27:08 Call Control - Maintenance MC330Ax DNI Line at location 06 1 04 removed
1556 2023/Feb/13 07:27:07 Call Control - Maintenance MC330Ax DNI Line at location 06 1 02 removed
1553 2023/Feb/13 07:25:19 Call Control - Maintenance The SYSTEM FAIL TRANSFER Zone at cab 06 shelves 01 , 01 RECONNECTED itself to the switch
1552 2023/Feb/13 07:25:11 Call Control - Maintenance ---------------+--------+-------+-------+-------+---------+---------+---------- Lines | 602 | 169 | 28 % | Major | 10 % | 20 % | 100 % | Trunks | 79 | 16 | 20 % | Major | > 0 % | 10 % | 100 % |
1551 2023/Feb/13 07:25:11 Call Control - Maintenance ---------------+--------+-------+-------+-------+---------+---------+---------- |Total In| Unavailable | Alarm Threshold Percentages Category | System | Total | % | Alarm | MINOR | MAJOR | CRITICAL
1550 2023/Feb/13 07:25:11 Call Control - Maintenance ------------ Current System Alarm : | MAJOR | viewed from Active ------------
1529 2023/Feb/13 07:24:59 Call Control - Maintenance The SYSTEM FAIL TRANSFER Zone at cab 06 shelves 01 , 01 ISOLATED itself from the switch
1522 2023/Feb/13 07:24:59 Call Control - Maintenance The link from MCECAAA MN3300 Controller at location 01 1 01 Active to MC312Ax Peripheral Cont I at location 06 1 16 is now in SCANNING mode.
1521 2023/Feb/13 07:24:58 Call Control - Maintenance MC312Ax Peripheral Cont I at location 06 1 16 removed
1510 2023/Feb/13 07:24:55 Call Control - Maintenance The link from MCECAAA MN3300 Controller at location 01 1 01 Active to MC312Ax Peripheral Cont I at location 06 1 16 is now in SCANNING mode.
1509 2023/Feb/13 07:24:55 Call Control - Maintenance MC312Ax Peripheral Cont I at location 06 1 16 removed

Under Maintenance Info Logs
Log Number 1505
Date 2023/Feb/13
Time 07:24:55
Source MSGLINKMGR
Description Could not detect the presence of the FIM connected unit for at least 1 second in link 0. ICB = 0x6

 

[Matty269]

Eyes

my alarms are not always from the same per node / fim card they rotate around between all the per nodes keep us posted on replacing the fim card