All phones rebooting on a teleworker site

[kwbMitel]

This one has me stumped and I already have Mitel support involved and so far we have not isolated the issue.

This is more of a hail mary as some of you guys have seen some strange s**t and this definitely qualifies

Hardware specs
- Multi-instance MCD server running 7.1
- MBG for SIP trunking and Teleworker gateway 8.1.26
- Dedicated 100Mb fiber internet at main site

Remote site
- Basic internet, specs unknown
- 10 X 5320e phones in teleworker mode

Symptoms
- Every phone resets after 4 hours
- Phones are not on the same schedule, any random phone might reboot within a 30 minute window but individually, every 4 hours like clockwork
- manually resetting a phone starts the timer again for that phone
- When the phone fails, the display indicates "Contacting Server" for anywhere from 1 to 15 minutes. Varies by instance
- Holding down the * (Star) key while in the Contacting Server state reveals the correct TW server address
- Phones will recover on their own but only after some time. Phones will recover immediately if rebooted manually
- It does not matter if the phone is in use or not, the phone will reset at 4 hours.

Troubleshooting steps
- Phones have been statically assigned to avoid DHCP issues
- A TW set was set up at a different location without issues
- The functional TW set was brought to site and it started rebooting the same as the others
- IT support has removed the firewall
- IT support indicates that there are no ICMP redirects configured on the site
- IT support has the network configured as a flat lan with only NAT in place before the internet
- The IP Phone Analyser has been deployed at the site and logs provided to Mitel
- Tug logs and MCD maintenance logs have been provided to Mitel

Steps planned to be taken
- Take a LTE wireless modem to site to connect a phone to the internet via a completely different service
- Pray, offer sacrifices, hop on one foot while spinning in place, etc

If you've read this far, thank you. Any suggestions?


**********************************************
What's most important is that you realise ... There is no spoon.

 

[wireman50]

I seem to remember something like this being reported on Cisco switches and turning off something on the port cured it.....but uselessly I can't remember what it was or even if it was this site or the other..... Sorry not really helpful....

 

[kwbMitel]

@Wireman50

Yes there was/is a known issue with ICMP redirects causing similar issues. That factor has already been eliminated.

**********************************************
What's most important is that you realise ... There is no spoon.

 

[Billz66]

I would get them to use a different POE Switch or an injector direct connected to the internet router

this will help identify whether its a problem with the switch


If I never did anything I'd never done before , I'd never do anything.....

 

[kwbMitel]

@Billz66

I'm not sure if the internet connection would have an alternate port and this doesn't smell like a POE issue as the phones are resetting at different times

Regardless, we are going to install a mobile internet service and POE injector to take 1 phone completely off of the existing equipment.



**********************************************
What's most important is that you realise ... There is no spoon.

 

[aavakian92]

Coincidentally enough we are having the same exact issue.
we just switched ISP's and went from a 20 MBit to 100 MBit connection.
we changed the TW IP on all the teleworker phones in the city around the main office and every hour or so ALL the Teleworker phones at the same time timeout and reconnect....
we have interfaced with our phone provider and our new ISP to no avail...
we are read for sacrifies and jumping in circles on one leg too...

 

[Billz66]

its just a means to eliminate the switch , next step would be change / reconfigure the router if it failed on poe injector
as per usual you'll know what it is when you know what it is , until then you just have to eliminate any suspects

If I never did anything I'd never done before , I'd never do anything.....

 

[wireman50]

Could it be a DHCP issue? I again remember some issue with the lease time being set down to a short time and for some unknown reason the phones were not getting the same DHCP re issued so were rebooting to get the new allocation working....setting the DHCP to a week masked the issue enough for peeps not to notice it.

 

[kwbMitel]

@Wireman50 - No, not DHCP, we statically assigned several phones with no change in symptoms.

@Billz66, installing the wireless internet service and placing a phone separate from the network has bypassed the issue. As you might expect, the customer now wants us to tell them what they need to fix. I sure wish I worked in IT where I could sit back and let everyone else do my job for me.

**********************************************
What's most important is that you realise ... There is no spoon.

 

[RAMitel]

I have come across issues with certain firewalls that have security services enabled such as gateway AntiVirus/Anti Spyware/content filters that will interfere with Mitel phones but as you state IT has removed the firewall, in this case am I correct in that the 5320's are connected directly to the ISP equipment? If so what router have they provided and is there a switch between the 5320 and the ISP router? Also have you tried wireshark on a phone to try and diagnose what is happening when it loses connectivity.

 

[kwbMitel]

@RAMitel - I have been told there is no firewall and only basic NAT routing. I suggested the possibility of Security/anti-virus without any response. We obtained traces on site using the IPAnalyzer tool from Mitel and Mitel has those traces but has not provided any feedback as yet.

**********************************************
What's most important is that you realise ... There is no spoon.

 

[aavakian92]

An update from us....
Upon further troubleshooting....
i plugged a laptop into our voice switch and its automatically configured DNS setup was all wrong and i coudldnt access the internet....
i told my boss who reprogrammed our cisco router when we changed ISP's and he sounded like he forgot to update some DNS settings for the Voice network we use. 192.168.10.x
hopefully after he changes that we will be back to normal.

 

[Billz66]

aavakian92
not sure that dns will help with TW , Minet phones use Ip address not dns for their connection


If I never did anything I'd never done before , I'd never do anything.....

 

[sarond]

If possible I would try a different switch.

Otherwise can you move all the services to the wireless connection and test for a day.

 

[libellis]

Phone reboot would occur if port speed is re-negotiated with the switch. Can't explain why this would occur with such periodicity (or need to reoccur at all), but maybe worth a look at switch and phone port speed settings/options.

 

[Jakt99]

KwbMitel,

Probably not applicable, but just putting this out there....

Check PoE power limitations and configuration on switch. If the device is manageable, you should be to get some stats from the device.
Perhaps the switch is shutting down a port(s) to prevent overload.

I have experienced some really weird stuff on "non enterprise grade" products that shut down ports (Aso called overload limit detected),
PoE device goes down, switch then re-applies power again because it sees a device attached to the port (no more overload).
What type / model switch is installed at site?

 

[kwbMitel]

@Jakt99

The symptoms do not indicate power issues. The phone never loses power. It first goes into a state saying contacting server then after 10 minutes, it does a soft reboot.

**********************************************
What's most important is that you realise ... There is no spoon.

 

[bribob]

What's the model number of the switch?
What type of 'NAT' device are they using? ** This is my first concern, depending on the type of traffic being passed through it **
Can the NAT device be configured to prioritize traffic to the TW Public IP?
What type of WAN connection is it, and is that link reliable? (read: not delivered via microwave)


-b

 

[aavakian92]

An update for you guys.
It was a problem with our AMC IP.
Our Mitel Partner "forgot" to tell us we had to de-activate then reactive the amc service to get it to pickup the new IP