Remote Phones over TLS, R11.1.3.1 - Soft Phones Log in, J100's Aquiring Service 5

[dsm600rr]

Hello All,

So we have a strange one here that we have tested for HOURS and cannot get remote J100's to log in.

So we have a Virtual Machine Running in a 11:11 Cloud, using TLS (no SBC) and NO IP SSL Certificates.

Remote Phones Log in OK on TCP

Remote Phones on TLS hang at "Logging in - Verifying Credentials" and then finally "Acquiring Service"

Avaya IX Workplace logs in fine on TLS

We tested phones both locally at the client, at out office and at home - all the same issues.

I can see the Remote Phone grabbing its files via Monitor.

We have quite a few other sites with the exact same setup and working fine. The ONLY difference we can see, is on this particular site, when we upgraded to R11.1.3.1 Build 34, Web Manager shows "Upgrade failed.: - Upgrade do stage failed to finish" - Of course Avaya says this is OK and to Ignore, however is it really?



ACSS / ACIS

 

[Shaun E]

Is there anything in the phone logs? (Admin -> Log -> Boot Logs).

Have you uploaded the necessary CA for No-IP SSL and updated "SET TRUSTCERTS" in your 46xxspecials file?

To me at first glance it seems like it could be a certificate trust issue. I believe IX Workplace uses the operating system trust store by default so it's entirely likely that's working as it's already trusting the certificate - but J100 sets you need to explicitly trust the root CA.

 

[dsm600rr]

Shaun E:
Thank you for chiming in.

I do not see anywhere (Admin -> Log -> Boot Logs) to actually view a log. The phone was set to: "log level: Errors".

I had wireshark running, and I did not see a TLS Handshake at all with the J100s - admittedly I am very green with wireshark, however I am pretty confident, on the previous install with the current filters I saw the handshake, so I am going to re-test tomorrow and see if I see the handshake on a known good working system with the same filters as today's test.

We are using the auto generated 46xxsettings.txt file.

Like mentioned, we have this same configuration, basically the same SSL Cert (With obvious FQDN/IP Updates), same software version (R11.1.3.1) all running in the same cloud data centers.

We can get to the FQDN/IPO and its showing as a secure TLS connection.

We can get to the FQDN/46xxsettings.txt

We can get to the FQDN/J100Supgrade.txt

I am seeing both:
SET TRUSTCERTS "Root-CA-0210033A.pem"
SET TRUSTCERTS WebRootCA.pem


ACSS / ACIS

 

[derfloh]

I would merge all certificates in one file and place it in the IP Office file management as WebRootCA.pem


IP Office remote service
IP Office certificate check
CLI based call blocking
SCN fallback over PSTN

 

[dsm600rr]

derfloh:

What would you change and/or add to the below process?

What I did was create an openssl.cfg like such:

Ran the following commands:

Downloaded the PEM Chain after pasting the .csr into no-ip

Ran the following commands:

And finally uploaded the .p12 to the IPO:

ACSS / ACIS

 

[telfire]

DSM,

If you browse to your system

https://IP

or FQDN/46xxsettings.txt and the look at the SET TRUSTCERTS line - what does it say there?
Then browse to

https://IP

or FQDN/whatever the file name was for the SET TRUSTCERTS (Mine is

https://FQDN/WebRootCA.pem

- the J-Series specifically look at this line on the 46xxsettings and then download said certificate to be in the phone "Trusted Certificate Store" - open that file after you web browse to it and convert to a .crt, then open that and make sure it is the RootCA used to create your Identity cert that you applied as a .p12 to the IPO Server. Like derfloh stated, you sometimes have to combine the root and identity into a single file, rename that to WebRootCA.pem and then upload that to Embedded File Manager system directory. You could change the name on the SET TRUSTCERTS line to what ever you want and then upload that created file to Embedded File Mgmt if you wanted to leave the existing auto generated WebRootCA un adulterated.

APSS-Midmarket
ACIS-Midmarket
ACSS-Midmarket
ACDS-Midmarket

 

[dsm600rr]

telfire:

Those are all good questions, however we had to revert them back to the modified 46xxsettings file to get the phones to re-register (as TCP) for business this morning (this is programming before I got a hold of the system).

This makes testing almost impossible, so I am not sure when I will be able to get back in, deleted the modified 46xxsettngs.txt in favor of the autogenerated and report back.

I guess my biggest thing is why are other sites, with the EXACT same Server config, software release and everything else working at all the other sites.

All sites are hosted in 11:11 Clound Data Center

All Sites are Server Edition, R11.1.3.1

All Sites are using J100's

All sites are using the same no-ip certificate, configured the same way.

I am a bit lost on "sometimes have to combine the root and identity into a single file" - Wouldn't downloading the "PEM Chain - Includes all necessary certificates in one file" from no-ip be "combining the root and identity into a single file"?

I appreciate all the help Gents!

ACSS / ACIS

 

[dsm600rr]

I did go to one of the working systems on the autogenerated 46xxsettings.txt just to see what shows there.

It has two lines for: "SET TRUSTCERTS":

SET TRUSTCERTS WebRootCA.pem
SET TRUSTCERTS "Root-CA-0210033A.pem"

I set up a remote phone here at my office to register to this working system and it looks like it grabs the WebRootCA.pem and not the "Root-CA-0210033A.pem"

11:11:41 606119657mS HTTP: Secure Rx Src: XXX.XXX.XXX.217(19346)-(411) MY PUBLIC IP
GET /WebRootCA.pem HTTP/1.1
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0) AVAYA/J169-4.0.10.3.2 (MAC:c81feaa0e17e)
11:11:41 606119659mS FILESYS: (20803) FilesysReader(Read): /opt/ipoffice/system/primary/WebRootCA.pem (AutoGenerate: Enabled) (Total = 1)
11:11:41 606119659mS FILESYS: (20803) FilesysReader: LastStateDuration(0) SetState(FS_UNKNOWN->FS_BEGIN): /opt/ipoffice/system/primary/WebRootCA.pem
11:11:41 606119659mS FILESYS: FileCache(f6147600): Created Cache: /opt/ipoffice/system/primary/WebRootCA.pem (Total = 5)
11:11:41 606119659mS FILESYS: (20803) FilesysReader: Using Cache: /opt/ipoffice/system/primary/WebRootCA.pem (Usage = 1)
11:11:41 606119659mS FILESYS: (20803) FilesysReader: LastStateDuration(0) SetState(FS_BEGIN->FS_OPEN): /opt/ipoffice/system/primary/WebRootCA.pem
11:11:41 606119659mS FILESYS: (20803) FilesysReader: Master Cache Freed for Virtual File: /opt/ipoffice/system/primary/WebRootCA.pem (Usage = 0)
11:11:41 606119659mS FILESYS: (20803) FilesysReader: LastStateDuration(0) SetState(FS_OPEN->FS_SENDING): /opt/ipoffice/system/primary/WebRootCA.pem
11:11:41 606119659mS FILESYS: (20803) FilesysReader: Send File: /opt/ipoffice/system/primary/WebRootCA.pem
11:11:41 606119660mS FILESYS: (20803) FilesysReader: SendConnectIndication: /opt/ipoffice/system/primary/WebRootCA.pem Size(1294) Attributes(0) TotalSize(1294) Range(1)
11:11:41 606119660mS FILESYS: (20803) FilesysReader: Percent(100): /opt/ipoffice/system/primary/WebRootCA.pem
11:11:41 606119660mS FILESYS: (20803) FilesysReader: LastStateDuration(1) SetState(FS_SENDING->FS_COMPLETE): /opt/ipoffice/system/primary/WebRootCA.pem
11:11:41 606119660mS FILESYS: FileCache(f6147600): Deleted Cache: /opt/ipoffice/system/primary/WebRootCA.pem (Total = 4)

Comparing both the downloaded "WebRootCA.pem" and "Root-CA-0210033A.pem" converted to .crt appear the exact same when viewed, I am not sure if they are correct, what I see is below:

ACSS / ACIS

 

[Ekster]

What type of certificate are you using? it needs to be a UC/Multi-SAN certificate with the SAN's containing the FQDN and the domain. E.G.

DNS Name = voice.mydomain.com
DNS Name = mydomain.com

The CA certs need to be for the certificate you use not the Avaya ones. From the above screen shots it looks like a Digicert certificate, look in the "view" certificate option for the certificate path, this will show you the CA certs you need.

Digicert Ca's:
DigiCert Global G2 TLS RSA SHA256 2020 CA1
DigiCertGlobalRootG2

“Some humans would do anything to see if it was possible to do it.
If you put a large switch in some cave somewhere, with a sign on it saying 'End-of-the-World Switch. PLEASE DO NOT TOUCH'.
The paint wouldn't even have time to dry.”

Terry Pratchet

 

[dsm600rr]

Ekster:

No-IP / DigiCert

View is greyed out.

ACSS / ACIS

 

[Ekster]

If view is greyed out then you don't have a certificate loaded, that might be your problem.

“Some humans would do anything to see if it was possible to do it.
If you put a large switch in some cave somewhere, with a sign on it saying 'End-of-the-World Switch. PLEASE DO NOT TOUCH'.
The paint wouldn't even have time to dry.”

Terry Pratchet

 

[dsm600rr]

Ekster: That is from a site with remote j100's working on TLS currently. I am able to view the certificate from the FQDN and verify it:

ACSS / ACIS

 

[Richard10d]

I have a cert like Ekster, but get my cert from a vendor who will also include the IP address. I would recommend letting IPO generate the 46xsettings.txt and use 46xxspecials.txt for any customization.

For many customers I use a 46xxspecials.txt file like the following, it has more in it than is needed you could make it quite a small file in reality, but I like having the flexibility of all the IF statements. I tell the phones not to be so uptight about the cert.

[pre]IF $MODEL4 SEQ 1603 GOTO 16XXSPECIALS
IF $MODEL4 SEQ 1608 GOTO 16XXSPECIALS
IF $MODEL4 SEQ 1616 GOTO 16XXSPECIALS
IF $MODEL4 SEQ 9620 GOTO 96XXSPECIALS
IF $MODEL4 SEQ 9630 GOTO 96XXSPECIALS
IF $MODEL4 SEQ 9640 GOTO 96XXSPECIALS
IF $MODEL4 SEQ 9650 GOTO 96XXSPECIALS
IF $MODEL4 SEQ 9608 GOTO 96X1SPECIALS
IF $MODEL4 SEQ 9611 GOTO 96X1SPECIALS
IF $MODEL4 SEQ 9621 GOTO 96X1SPECIALS
IF $MODEL4 SEQ 9641 GOTO 96X1SPECIALS
IF $MODEL4 SEQ J129 GOTO J1X9SPECIALS
IF $MODEL4 SEQ J139 GOTO J1X9SPECIALS
IF $MODEL4 SEQ J169 GOTO J1X9SPECIALS
IF $MODEL4 SEQ J179 GOTO J1X9SPECIALS
IF $MODEL4 SEQ K165 GOTO K1XXSPECIALS
IF $MODEL4 SEQ K175 GOTO K1XXSPECIALS
GOTO GENERALSPECIALS
# 16XXSPECIALS
GOTO GENERALSPECIALS
# 96XXSPECIALS
GOTO GENERALSPECIALS
# 96X1SPECIALS
SET TLSSRVRID 0
SET TLSSRVRVERIFYID 0
GOTO GENERALSPECIALS
# J1X9SPECIALS
IF $SIG_IN_USE SEQ H323 GOTO J1X9H323SPECIALS
SET TLSSRVRID 0
GOTO GENERALSPECIALS
# J1X9H323SPECIALS
GOTO GENERALSPECIALS
# K1XXSPECIALS
GOTO GENERALSPECIALS
# GENERALSPECIALS
# GROUP_SETTINGS
IF $GROUP SEQ 1 GOTO GROUP_1
IF $GROUP SEQ 2 GOTO GROUP_2
IF $GROUP SEQ 3 GOTO GROUP_3
IF $GROUP SEQ 4 GOTO GROUP_4
IF $GROUP SEQ 5 GOTO GROUP_5
GOTO END
# GROUP_1
GOTO END
# GROUP_2
GOTO END
# GROUP_3
GOTO END
# GROUP_4
GOTO END
# GROUP_5
GOTO END
# END[/pre]

 

[dsm600rr]

Ekster:

"It needs to be a UC/Multi-SAN certificate with the SAN's containing the FQDN and the domain."

This may be part of the issue. I am able to generate the CSR and get the certificate with the CSR Containing the FQDN as DNS.1.

If I add: DNS.2 = as the domain to the CSR and load it to no-ip/digicert, I get: "Invalid Subject Alternative Name. Must be a subdomain of the common name"

ACSS / ACIS

 

[Ekster]

If you are only getting the FQDN then you need to change the certificate to a multi SAN so you can also have the domain too.

From experience everything will seem OK, files download using HTTPS and the Workplace client will work too, but the J100 phones need that multi-SAN cert to connect using secure SIP (5061).

“Some humans would do anything to see if it was possible to do it.
If you put a large switch in some cave somewhere, with a sign on it saying 'End-of-the-World Switch. PLEASE DO NOT TOUCH'.
The paint wouldn't even have time to dry.”

Terry Pratchet

 

[dsm600rr]

Ekster: That may be true, however we have this working already with just the FQDN at multiple sites. We use the FQDN in the IPO for both: SIP Domain Name and SIP Registrar FQDN, with only the FQDN listed on the Certificate. We own the clients domain/FQDN for this. If we indeed do NEED a multi-SAN cert, and perhaps we just got lucky? We can look into that, but we will need 100% confirmation as this will change a whole lot.

I got bored and set up a J169 with a option242 pointing to one of the working systems on TLS and port mirrored a packet capture of the phone registering. I can see the TLS Handshake and all looks good from what I can see. Like I said, the IPO/Certificate on this Working system outlining the packet capture is the same setup as the system we are having issues with.

ACSS / ACIS

 

[Ekster]

Check a know good certificate and look under Details/Subject Alternate Name, you should see the FQDN and the domain entries at least:

Then compare it to the one that doesn't work.

“Some humans would do anything to see if it was possible to do it.
If you put a large switch in some cave somewhere, with a sign on it saying 'End-of-the-World Switch. PLEASE DO NOT TOUCH'.
The paint wouldn't even have time to dry.”

Terry Pratchet

 

[dsm600rr]

Ekster:

If I grab the cert from the web FQDN I see the SAN (Just the FQDN):

If I browse to the

https://fqdn/WebRootCA.pem

and convert to .crt - I do not see any SANS's. This is on a Working system using TLS / Digicert Certificates (same system as the above packet capture).

ACSS / ACIS

 

[derfloh]

WebRootCA.pem is the CA cert. It does not need SANs. You have to check the server certificate.

IP Office remote service
IP Office certificate check
CLI based call blocking
SCN fallback over PSTN

 

[dsm600rr]

Found the error running a PCAP - Thoughts?

ACSS / ACIS