Early morning MCD losing heartbeat to phones.

[vlevalois]

Have a customer that has had this happen only once, but I've seen it before on one other system and it's a mystery as to what could be causing it.

Basically, the MCD (5.0 SP1 on an MXe with all 5320 IP phones) reports this for all phones early in the morning, before 7am:


Log Type Software
Log Number 1392
Severity Warning
Date 2012/Jan/14
Time 06:45:10
Source Heartbeat Server
Description ICP has lost contact with (192.168.2.119 (08-00-0F-5E-B0-0E)), network driver cluster pool free: 3003 and low water mark: 2805
Module Main
File Name and Line Number HeartbeatRecvTimeoutHandler.cpp;273

This happens to all phones, all in succession. The MCD is located on a voice VLAN with DHCP redirect for the phones off of a Cisco router.

The next event after this happening is:

Log Type Software
Log Number 1393
Severity Warning
Date 2012/Jan/14
Time 09:20:41
Source SecurityMgr
Description RequestMatchingProfileData() System is running with GDM Mode. Attempting to retrieve user profile.
Module Main
File Name and Line Number SecurityManager.cpp;1281

Several of these types of errors appear. The system had to be rebooted (supposedly) in order for everything to work again. We suspect a routing/network issue? The system has a PRI also.

Any ideas as to what could be happening?

Thanks.

 

[DoubleUT]

The problem might already happened before this LOG entry. This is an issue that needs reporting back to Mitel asap.

 

[LoopyLou]

Anything going on within the network at that time? Backups?

I'd tell you a UDP joke but I'm afraid you won't get it. TCP jokes are the best because you always get them.

 

[vlevalois]

Nothing happening on the network according to the IT folks at the customer site and some logs they sent me. We're going to report this for sure with Mitel.

Thanks.

 

[LoopyLou]

What about power? Are the sets powered at the desk or from the closet? If at the desk is there some type of generator test run in the AM? If PoE are the layer 2 switches backed up? Same thing re generator test.

I'd tell you a UDP joke but I'm afraid you won't get it. TCP jokes are the best because you always get them.

 

[dryaquaman]

I've seen something similar. Every morning at same time the system would loose connectivity to all phones. Then they'd all come back up.
Customer eventually traced it to a Cisco switch rebooting every day. (I've no clue how this would be set up)

Dry Aquaman

http://www.university-music-on-hold.com

 

[vlevalois]

Well is hasn't happened since, only the one time, but the customer is sensitive enough to need an explanation... They pulled POE logs and the switch was up the whole time. The phones had power and showed "Waiting for ACK... ICP 192.168.1.10" (the latter being the ICP's address).

At this point we're waiting for Mitel's analysis of the diag file.

 

[LoopyLou]

Nothing in the MCD to indicate a reboot happened?

I'd tell you a UDP joke but I'm afraid you won't get it. TCP jokes are the best because you always get them.

 

[KurpeusLondon]

Is there a VPN between the two controllers ? It could have been that the IKE session expired and the VPN took some time to recover (although new keys should be negociated before timer expires)