Avaya 9611G H323 6.8.3.2Y EAP-TLS SCEP renewal

[avayaguy23]

I am deploying 802.1x on 9611G phones running H323 6.8.3.2Y firmware. I have it all working but I have questions about the SCEP renew process. I currently have the cert identity policy to be valid for a day. The renewal process kicks in at 90% or about 21 hours. The phone goes offline for a couple days but is connected to a non 802.1x enabled network port so it can perform the SCEP process. It never performs the SCEP renewal process. ClearPass is saying the phone is using the old expired identity cert when connected to an 802.1x enabled network port. We need to factory reset the phone in order for it to generate a new identity cert. Is this normal behavior? I'm pretty sure it is based on the fact that the phone has 802.1x enabled but not on the network port.

DOT1X – set to “2 – EAPOL multicast passthrough is disabled”
DOT1XSTAT – set to “1 – enable supplicant for multicast and unicast EAPOL messages”
DOT1XEAPS – set to TLS
this will enable EAP-TLS authentication
DOT1XWAIT - set to 1

Configure
MYCERTCN – set to $MACADDR
configures the identification method to using the phone’s MAC address

MYCERTURL – to include the URL of the CA server where the identity certs are stored
e.g.:

https://x.x.x.x/certsrv/mscep/mscep.dll

The phone with download the certificate using the SCEP process

MYCERTKEYLEN – 2048

 

[jimbojimbo]

Saw this back in December 2020. Pretty sure it is a software bug. There is a way to get around it but I forget what the solution was on the data network side. I think it was a timer or auth type issue. I suggest you open a ticket with Avaya.

Quick fix for the phones is to temporarily disable 802.1x so the phone can successfully get onto the network and pull a new certificate then reenable 802.1x.

 

[avayaguy23]

Thanks for the response. I assumed it was a bug and has been for a while. I tried software ranging from 6.66 to the latest 6.8.5.1. I opened a ticket with avaya a month ago and tier4 is dragging their feet (shocker). Apparently avaya only has one person in the entire company that understands 802.1x. Hopefully I’ll have a response from them in the next week. I can’t roll out 802.1x to several thousand phones until they provide working firmware.

 

[avayaguy23]

The responses from Avaya have been extremely slow but we have made some progress. They recommended I change the cert life from 1 day to 16 days, but the cert was still renewing at 12 hours when the MYCERTRENEW setting was set to 1. We changed the MYCERTRENEW to 50 and it seems to be working better. This works out to renew at 8 days so we will see if that holds true next week. I still have my concerns as there seems to be a bug with the software. The developers are looking into this further in their lab and potentially working on a firmware patch.

 

[avayaguy23]

Update- Avaya developers admit it appears to be a bug. Hopefully I'll get working firmware by the end of the year.

 

[avayaguy23]

Avaya provided us with firmware: 96x1-IPT-H323-R6_8_5_52-111621 to resolve the cert renewal issue.

 

[AuraPS]

I have the same issue,

A 90% certificate expiry warning has flashed up on all our phone displays (all 1500 of them) and caused widespread panic.

SET TRUSTCERTS CertChain.pem
SET MYCERTCN $MACADDR
SET SCEPPASSWORD xxxxxx
SET MYCERTURL

http://xxxxxxxxx.xxx/certsrv/mscep/mscep.dll

SET MYCERTKEYLEN 2048
SET DOT1XSTAT 2
SET DOT1X 1
SET DOT1XEAPS TLS


I know I will need to upload a new CA cert with the extended expiry date to the utility server but will the NDES server renew the phone certs automatically?

TIA

 

[avayaguy23]

The MYCERTRENEW setting doesn't work correctly. The phone initiates the renewal which is where the 90% comes into play. Avaya thought they fixed it with the above mentioned firmware but I am on firmware revision 3 and still not fixed. If the phone does not renew at 90% and the cert expires, they will be fine until the phone reboots.

 

[AuraPS]

Thanks for the response, have you (or Avaya) any suggestions on how best to get these 1500 phones to update their certificates without us having to

1. Disable network authentication on all our switches
2. Factory reset all our phones
3. reset all phones to load up 46xxsettings file again
4. An additional reset to go through the whole SCEP exchange and enable 802.1x process again.
5. re-enable NAC on all our switches

This is becoming a major concern for us

If you can outline some steps that would be greatly appreciated

Many thanks

 

[avayaguy23]

I would recommend opening a ticket with Avaya. It took 3 months to even get the ticket to the developers. You are using H323 firmware correct?

 

[AuraPS]

Yes H323 - 6.8.3

 

[avayaguy23]

Yikes then yea you will run into the same issues I am facing. I definitely wouldn't trust their certificate renewal setting. Your above thought process is correct. The only other thought would be to increase your generated certificate life from say 1 year to 2 years so when you factory reset the phone it has an extended life. Another option would be to just let the certificates potentially expire and then deal with them as tickets come in. Unless you deployed all 1500 phones at the same time then that approach would be feasible. I honestly don't think Avaya will be fixing this issue in the next month or two. Realistically, it will be closer to 6 months and then you have to go through all the other validations in case Avaya broke something else along the way.

 

[avayaguy23]

Update: Avaya provided firmware 96x1-IPT-H323-R6_8_5_56-011722 that at least resolves part of the problem. It appears the certificates are renewing approximately every 15 minutes when I set the renew rate to 1% and the certificate is valid for a day. Avaya added the following setting to the 46xxsettings to address the issue: SET CMT_RENEWAL_OFFSET. I initially set the SET CMT_RENEWAL_OFFSET -7 (MST) which actually caused the certificate to renew every 14 hours. I had to change the setting to: SET CMT_RENEWAL_OFFSET 7. I was assuming that setting was based off of GMT (MST) which is -7.

I still do have concerns with daylight savings. Since March 13 will soon be around the corner I will be able to easily tell if daylight savings will be an issue or not. I assume it will be an issue based on what I found with the SET CMT_RENEWAL_OFFSET parameter. The parameter will need to be changed to 6 and all the phones in the environment will need to be rebooted. I am still talking with Avaya if they will resolve this proactively or if I will need to open up a GRIP/another ticket.

 

[avayaguy23]

Avaya is taking the stance of opening a GRIP for daylight savings support.

 

[avayaguy23]

Here is the final resolution. H323 firmware 6.8.5.2 will solve the EAP-TLS renewal with time zone offset. Avaya however denied to implement daylight savings support for H323. Per Avaya- The solution is to migrate to SIP firmware. So I guess the short answer is anyone wanting to implement 802.1x in their environment should migrate to SIP as Avaya refuses to support 802.1x on H323.

 

[Daudet]

Hi
i am having same issue, cnat get certificates for H-323 phone


avayaguy23 (Systems Engineer)(OP)20 Jan 22 16:20
Update: Avaya provided firmware 96x1-IPT-H323-R6_8_5_56-011722 that at least resolves part of the problem.

Is this a custom FW you got ?
Any possibility to have it do i can test in case i have a faulty FW.

I am using this one 96x1-IPT-H323-R6_8_5_2U-033122 and no luck to get my certificates with H.323 phone

we try with a SIP phone with S96x1_SALBR7_1_15_0r14_V4r83 and it works.



Daniel Audet
IT Technical Advisor Senior | Infrastructure, Evolution & Services|
Intact Financial Corporation
Montreal, Quebec, Canada

 

[Daudet]

Hi everyone
Here is more info on my issue.

H.323 Phone is at FW 6.8.5.2 and I never reach the SCEP server to get certificated
We used 6.8.5.2 until Avaya told us that 6.8.5.3 was out recently which we have try on a live trouble shooting session with Avaya Engineer last week. But result is the same.
96x1-IPT-H323-R6_8_5_2U-033122 even try 96x1-IPT-H323-R6_8_5_3U-061422 as suggested by Avaya.

We even enable the debug and ssh to the phone and Avaya got in the phone thru my laptop and logged in with craft because it send a challenge then with WinSCP we downloaded the PCAP file
and send it to Avaya to analyze.
Here is the response we got after.

Our SME has check the PCAP trace and found that there might be issue on the DHCP process. You need to have scope 242 configured in the DHCP server. It is potentially a network issue. We need to start with billing process. If at the end, it is not network issue, we will not bill anything. However, if it is network issue, from this point, all time and material we spend on the case will be billed.


Well DHCP scope have been there for the past 7 years and never changed.
242 Avaya Option 242 MCIPADD= xxx.xxx.xxx.xx,MCPORT=1719,HTTPSRVR=xxx.xxx.xx.xx,VLANTEST=30
006 DNS Servers xxx.xxx.xxx.xx,xxx.xx.xx.xx
How can it be potentially a network issue. If nothing have changed.
What could on our network affect phone from not talking to SCEP server. Even mirroring the switch port of the phone with Wireshark we never see any attempt to reach any SCEP server IP



Here is our closed-circuit setup to test update
1. I have a port NON 802.1x
2. Plug a phone and in CRAFT menu Clear phone to factory
3. Upon reboot I stop the phone with * to program and setup to group 91 to use my 46xxsettings_91.txt, 96x1Hupgrade is set to update phone to 6.8.5.2 or 6.8.5.3 depend on testing . then we restart the phone . Here is what I get on the screen until I get the login (* to Prog, VLID=0, waiting for LLDP, Restart, * to Prog, VL=66, HTTPSRVR, 96x1, 46xx91, slamon and all cert 200 ok, Download TAR and update process, restart, * to Prog, waiting LLDP, restarting, * to Prog, HTTPSRVR, 96x1, 46xx91, contacting call server. Login)
4. Log back in craft and restart phone again and does not get to SCEP server.
5. Moving the phone to a 802.1x port and still does not get anywhere.


Here is my 46xxsettings
and using the same 46xxsetting for a SIP phone it does get the certificates and with h.323 is does not.

########### 46xxsetting_91.txt ################
SET MCIPADD xxx.xx.xxx.xx
SET HTTPSRVR xxx.xx.xxx.xx
SET BRURI

http://xxx.xx.xxx.xx/phonebackup

SET DNSSRVR xxx.xx.xxx.xx,xxx.xx.xxx.xx,xxx.xx.xxx.xx
##
##################### Certificates #############################
##
SET TRUSTCERTS slamon-cert-chain2.pem.txt,MyCieIssuingCA-Private-1.pem.txt,MyCieIssuingCA-Private-2.pem.txt,MyCieIssuingCA-Private-3.pem.txt,MyCieRootCA.pem.txt,default.cacert.pem.txt,MyHost.tad.inet.pem.txt
SET MYCERTURL

http://10.xxx.xx.xx/certsrv/mscep/mscep.dll

SET MYCERTCN "$MACADDR"
SET MYCERTDN /C=CA/ST=QC/L=MTL/O=IFC/OU=AVAYA_IPPhone
SET MYCERTKEYLEN 2048
SET MYCERTRENEW 99
SET MYCERTWAIT 0
SET SERVER_CERT_RECHECK_HOURS 1
SET CERT_WARNING_DAYS 1
SET CMT_RENEWAL_OFFSET "-4"
##################### DEBUG Activation #########################
## SET PROCPSWD xxxxx
##################### SLA Mon setup ############################
##
SET SLMSTAT 1
SET SLMSRVR xxx.xx.xxx.xx
SET SLMCTRL 1
SET SLMPERF 1
SET SLMCAP 2
##
##################### 802.1x Setup ############################
##
## SET DOT1XSTAT 2
## SET DOT1X 1
## SET DOT1XEAPS TLS
## SET DOT1XWAIT 0
## SET DOT1XEAPTLSONLYWITHCERT 1
##
##################### SIP Config ###############################
##
##
SET ENABLE_PPM_SOURCED_SIPPROXYSRVR 1
SET SIPDOMAIN "sip.xxxx"
## SET SIP_CONTROLLER_LIST
SET SIP_CONTROLLER_LIST xxx.xx.xx.xx:5061;transport=tls
SET CONNECTION_REUSE 1
SET ENABLE_PPM 1
SET CONFIG_SERVER_SECURE_MODE 0
SET ENABLE_AVAYA_ENVIRONMENT 1
SET STMPSRVR xxx.xx.xx.xx,yyy.yy.yy.yy
SET DSTSTART 2SunMar2L
SET DSTSTOP 1SunNov2L
##



Daniel Audet
IT Technical Advisor Senior | Infrastructure, Evolution & Services|
Intact Financial Corporation
Montreal, Quebec, Canada

 

[avayaguy23]

Avaya "resolved" the issue in 6.8.5.2 so I would recommend using that firmware. Did you check your utility server to see if the phone is properly downloading the certificates and 46xxsettings file?

 

[Daudet]

Yes I use this version 6.8.5.2 and we are using a Windows server with IIS for HTTPSRVR

Yes it does see the 46xxsettings and H96x

but for Certificates they are not downloaded from HTTPSRVR but form the SCEP server right?
As I explained sniffing the phone port with Wireshark et don't see the phone requesting certificates to SCEP server URL or IP.

Daniel Audet
IT Technical Advisor Senior | Infrastructure, Evolution & Services|
Intact Financial Corporation
Montreal, Quebec, Canada

 

[avayaguy23]

You are missing the following configurations. Its not gonna request certificates without it.
SET DOT1XEAPS TLS
SET DOT1XWAIT 1
SET DOT1X 1
SET DOT1XSTAT 2