Random Phones - Acquiring Service. IPO SE Remote Phones

[dsm600rr]

Hello all,

We have a Avaya IPO SE in a 11:11 Cloud with 5 Remote Locations.

One of the locations is having in issue with a few phones, the rest are registered. A handful are showing "Acquiring Service".

When I cleared the "VoIP Security" the down phones logged in. What could be causing the IPO to block a few random remote phones? They are all coming from the same Public at this particular location.

 

[john3voltas]

EDIT:
Sorry, i misinterpreted your message.
It seems you only have a single SE instance. What you seem to have is 5 different phone locations. If the remote phones on the other 4 locations don't support encryption (16xx) then it might still make sense. But if they don't, then my expired certificate theory doesn't make sense after all...

You consider it random. Well, I don't, otherwise they wouldn't all belong to the same location.
Are these J100 series phones? Like J139/J159 or of the sort?
If they are, *that* and the fact that they could register disabling VoIP security makes me think of certificates.
For instance, recently expired identity certificates on the SE instance that they all register to.
Either that or firewall issues that are preventing access to port 5061. They can crossover your firewall to port 5060 but they can't do the same to port 5061. That's another possibility.
But I'd immediately check the certificates under system security and check on their expiry date.
Regards

 

[dsm600rr]

@john3voltas
"It seems you only have a single SE instance. What you seem to have is 5 different phone locations.": Its a very Similar Setup to Powered by. IPO SE in a 11:11 Cloud, 5 locations.

"If the remote phones on the other 4 locations don't support encryption (16xx) then it might still make sense. But if they don't, then my expired certificate theory doesn't make sense after all...": Phones are J139's / J159's / J179's

You consider it random. Well, I don't, otherwise they wouldn't all belong to the same location: This location has 30 phones, 5 are down. 25 are not having any issues. Its definitely a location specific issue, however only effecting a handful of phones and the location.


Are these J100 series phones? Like J139/J159 or of the sort? J139's / J159's / J179's

If they are, *that* and the fact that they could register disabling VoIP security makes me think of certificates. Same cert has been in place for 2mo. DigiCert.

For instance, recently expired identity certificates on the SE instance that they all register to. Certificate set to expire Aug 18th 2025

Either that or firewall issues that are preventing access to port 5061. They can crossover your firewall to port 5060 but they can't do the same to port 5061. That's another possibility. Could be firewall, they have another company that handles that and of course everything "looks" ok. We are using Port 5061 on TLS.

But I'd immediately check the certificates under system security and check on their expiry date. Certificate set to expire Aug 18th 2025

Regards

 

[nnaarrnn]

First off- when did this begin? How long did they work beforehand?

Second- in SSA, Go to: System->VoIP Security->Blacklisted Addresses and see if the public IP of that location is listed, also check Blacklisted Extensions right above that.

If a user or phone hammers the call server with incorrect login info, the system could blacklist that public IP.


Also:
Make sure whoever is handling the firewall has any Deep Packet inspection (DPI-SSL in the SonicWall world) COMPLETELY disabled for that vLAN. It needs to be disabled for the Voice-vLAN->WAN access rule that allows that vLAN to get to the WAN (assuming no VPN tunnel exists here). I'd also have them show you screenshots proving SIP-ALG Transformations are disabled and that Consistent-NAT is enabled. (my line of thinking is that if these *did* work, but now don't, a firewall update/patch could have changed things unexpectedly)

 

[dsm600rr]

@nnaarrnn​

Thank you for taking the time to respond:

We switched them over to TLS 2.5 months ago and they have been fine for the first two months. They have IPO SE in a 11:11 VM w/5 locations using TLS (DigiCert) - no SBC. We have many other clients running the same everything with no issues.
All of a sudden a few weeks ago, one of the 5 locations lost all of their phones. They were all at "Acquiring Service". To make a long story short, we got IT on the horn, they said they would look into it and call me back. About 2 min later they all came back up, I called him to say whatever he did worked, and he told me he did nothing. We basically left it at that.

Fast forward to this week. First of all we have been have power outage all over (I'm in SC) from the Hurricane - so that isn't helping things.
Anyway, we get a call today, that another location (different from the first that happened a few weeks prior) has about 5 phones (out of 20 I believe) that are also showing "Acquiring Service". We get the firewall admin on the horn again - hes not seeing anything odd. So while they are talking, I go into SSA and just clear out all 3 "VoIP Security" Columns and then I see all of the 5 phones log in. This was earlier today when I made this post.

Now, in SSA I am seeing their public blocked for TLS 5061 under "Blacklisted Addresses". Why would the IPO be blocking this locations public? I'm wondering if that locations public gets blocked but some of the phones stay registered unless they reboot or something. With the below example, all of the phones were at "Acquiring Service" at this specific location. When I cleared this error, 20 or so phones logged back in. This was from my testing just now writing this response.

Also, they have an interesting public - thought I would note that. Starts with a 192 - not sure if things could be getting confused?


I will confirm your recommendations with their Firewall admin tomorrow am.
I went to add their public to the IPO's: VoIP > Access Control Lists > IP Whitelist however I had a few questions first.
- Why would just this locations Public be getting blocked in "Blacklisted Addresses"
- When I went to add this to the IP Whitelist, I wasn't able as it erred: "Maximum 20 IP Whitelist entries are supported" - I do not recall adding any however their are 20 in there apparently.

Thanks!

 

[Menniss]

We run a number of systems in the same configuration including some tenanted. (Cloud no SBC)

We don't often have an issue with blacklisting but when it happens for large site its very annoying especially if the customer cant find the phone causing the issue.

Blacklisted Addresses happens if a phone attempts to register with incorrect credentials, its done this based on the Public IP. at 10 incorrect it will block the IP.

The error in the screen shot looks more like a TLS error with incorrect formatting. We had this once when we switch a system that wasn't using TLS to using TLS. Most phone were fine they rebooted and too the new setting file. Some didn't and had to be defaulted.

Do you use DES?

On the whitelist, the system adds these automatically. Annoying that it's limited to 20, we remove the ones we don't care about and add ones that matter. I would be interested to know more about this function if anyone cares to share?

 

[dsm600rr]

@Menniss

I noticed the LAN2 (192.168.43.1) was in the IP Whitelist which is not being used so I deleted it and added their Public IP.

They could have a phone with the incorrect credentials trying to register from that IP over and over, however with how slow Avaya phones are - I don't really see that being the cause - but anything is possible.

This morning I saw the Public blocked again - yet all the phones were still logged in. I went ahead and Whitelisted it anyway.

The IPO has "TLS" Enabled on the LAN1 > VoIP Tab. All phones were reset to default when we cut them over to TLS.

I am not sure what DES is.

Thanks!

 

[derfloh]

Let a monitor trace run with default filters. You should see information why the IP is blocked there.

 

[Menniss]

I agree with derfloh Monitor is where you are going to find out what is causing the issue.

DES is Avaya's provisioning server we use it for all our cloud deployments make deploying the J100s much simpler.

Log in to Device Enrollment Service

des.avaya.com

 

[john3voltas]

DES is Avaya's provisioning server we use it for all our cloud deployments make deploying the J100s much simpler.

I was under the impression that DES was only available to Aura customers. And maybe AXP customers.
Is it available to SMB customers too?
Cheers

 

[Menniss]

@ john Yep we are only in the midmarket and use it all the time

 

[derfloh]

It is available for J-Series phones. Regardless of the user PBX.

 

[Travis Harper]

Hello all,

We have a Avaya IPO SE in a 11:11 Cloud with 5 Remote Locations.

One of the locations is having in issue with a few phones, the rest are registered. A handful are showing "Acquiring Service".

When I cleared the "VoIP Security" the down phones logged in. What could be causing the IPO to block a few random remote phones? They are all coming from the same Public at this particular location.

What is unique about these 5 phones? Is the network at that location converged? Are there 2 network drops at the desk. 1 for voice and 1 for data. Is it possible that those 5 phones are plugged into a port with a VLAN that does not have access to the IPO server? If all other phones at that location work, but these 5 don't, then he issue could be network related.

I create whitelists on my system to only allows specific devices with specific firmware, and sometimes this catches me. Could it be possible that those 5 phones are using a different firmware version than the other 25?

You can trouble shoot this issue by simply going into the IP Office System Settings, under (LAN1/LAN2 <- whatever the registrar is set to use), Allowed SIP User Agents, and set it all. If those 5 phones come up, then you know it has something to do with access controls.

 

[john3voltas]

Lack of available licenses can also end up in acquiring services.
Please check the alarms in SSA and make sure there are no license alarms for Avaya IP Endpoints.

Doh!! Forgot you mentioned this works when you disable security...
Never mind.

 

[Travis Harper]

Hello all,

We have a Avaya IPO SE in a 11:11 Cloud with 5 Remote Locations.

One of the locations is having in issue with a few phones, the rest are registered. A handful are showing "Acquiring Service".

When I cleared the "VoIP Security" the down phones logged in. What could be causing the IPO to block a few random remote phones? They are all coming from the same Public at this particular location.

Are there entries in "Allowed SIP User Agents" and if so, the User Agents of those 5 phones different from the other 25 in any way.. including Firmware version.

 

[john3voltas]

Guys, the phones work with security disabled in the VOIP tab of SYSTEM > LAN1. It can't be user agent.
Firmware is not impossible, of course. But would need to be a firmware bug.
I'm fully convinced that this is a certificate issue. The certificate has been updated recently. IIRC you mentioned 2 month ago, right? And that's about the same time when the issue with these 5 started?
All phones received the new certificate, all but these 5 did not for some reason we can't tell right now.

@dsm600rr , have you tried resetting the phones? A full reset forces the phone to reacquire the certificate.
Or maybe you need to erase the old certificate on those 5 and reupload the new certificate to those phones.
EDIT: A reset might not be enough. You may need to reset and upload the cert to the phones manually.

 

[Travis Harper]

Guys, the phones work with security disabled in the VOIP tab of SYSTEM > LAN1. It can't be user agent.
Firmware is not impossible, of course. But would need to be a firmware bug.
I'm fully convinced that this is a certificate issue. The certificate has been updated recently. IIRC you mentioned 2 month ago, right? And that's about the same time when the issue with these 5 started?
All phones received the new certificate, all but these 5 did not for some reason we can't tell right now.

@dsm600rr , have you tried resetting the phones? A full reset forces the phone to reacquire the certificate.
Or maybe you need to erase the old certificate on those 5 and reupload the new certificate to those phones.
EDIT: A reset might not be enough. You may need to reset and upload the cert to the phones manually.

Sorry, I misread. I thought you confirmed the phones all got the certs. If they don't have the certs then that is 100% the issue. Factory reset the phone and let it pick them up again on reboot...as long as you are using the auto generated 46xxSettings.txt file.

 

[john3voltas]

as long as you are using the auto generated 46xxSettings.txt file.

Precisely.
My point is that either those 5 phones couldn't/can't reach the file server for some odd reason, or they are not all picking up the 46xxsettings.txt due to some weird firewall issue/rule.
Those phones should be reset and the OP should make sure that the phones pick up the 46xxsettings.txt. I remember some place that can be checked. Monitor app? Can't recall.
Cheers

 

[Travis Harper]

Precisely.
My point is that either those 5 phones couldn't/can't reach the file server for some odd reason, or they are not all picking up the 46xxsettings.txt due to some weird firewall issue/rule.
Those phones should be reset and the OP should make sure that the phones pick up the 46xxsettings.txt. I remember some place that can be checked. Monitor app? Can't recall.
Cheers

In that case navigate to the address of your IPO/46xxsettings.txt file in a web browser and copy it. Save it to a local web server those phones can access, and then manually change the file server address on the phone to the new webserver hosting that 46xxsettings.txt file.

 

[derfloh]

In that case navigate to the address of your IPO/46xxsettings.txt file in a web browser and copy it. Save it to a local web server those phones can access, and then manually change the file server address on the phone to the new webserver hosting that 46xxsettings.txt file.

That will probably not help. The phones need the settings file and they need HTTP(S) connection to IPO for call control, call log, phone book and further services.