SMGR 6.3 Renew certificate failed

[hcmbmt]

Hi all,
The cert of SMGR 6.3 is expired and I cannot login to SMGR web admin.
To renew, I followed the PSN003661u to renew the certificate with below steps:
1. Copy "CertificateRenewalUtility_v2.bin" to tmp folder of SMGR
2. Run "#chmod +x CertificateRenewalUtility_v2.bin"
3. Run "#sh CertificateRenewalUtility_v2.bin"
but it always failed. The log as below:

Do you want to apply patch? (y/n). y
Thu Feb 29 10:47:15 ICT 2024 : Starting Patch application
Thu Feb 29 10:47:15 ICT 2024 : Unpacking binary files...
Zip successfully extracted
Thu Feb 29 10:47:15 ICT 2024 : Taking backup of certificates to //var/log/Avaya/CertificateGeneration_29_02_24_104645/
Service id apache_load_balancer has internally signed certificate
Service id container_tls has internally signed certificate
Generating intermidiate certificate for renewal
Generating CA Certificate
CA Certificate generated sucessfully
Adding CA certificate to default_tls_outbound_truststore.jks
CA certificate added sucessfully to default_tls_outbound_truststore.jks
Adding CA certificate to default_tls_outbound_truststore.jks
CA certificate added sucessfully to default_tls_outbound_truststore.jks
Generating Jboss certificate
Jboss certificate generated sucessfully
Generating Postgres certificate
Postgres certificate generated sucessfully
Return Code: 0(Java Execution completed)
Thu Feb 29 10:47:17 ICT 2024 : Updating System manager database with the temporary certificate
Thu Feb 29 10:47:17 ICT 2024 : Changing execution modes of keystore files
Thu Feb 29 10:47:17 ICT 2024 : Changing ownership of postgres certificates to postgres user
Thu Feb 29 10:47:17 ICT 2024 : Restarting System Manager database service
Thu Feb 29 10:47:21 ICT 2024 : System Manager database certificate updated successfully
Thu Feb 29 10:47:21 ICT 2024 : Starting System Manager services ....
Thu Feb 29 10:49:50 ICT 2024 : Waiting for System Manager services to start..
Thu Feb 29 10:49:50 ICT 2024 : Sleeping for 6 minutes, please be patient DO NOT KILL THE PROCESS
SThu Feb 29 10:55:50 ICT 2024 : Running the trust initializer script
Thu Feb 29 10:55:53 ICT 2024 : Sleeping for 2 minutes and trying again. Try 1 of 10
.......
Thu Feb 29 11:17:52 ICT 2024 : Sleeping for 2 minutes and trying again. Try 10 of 10
Thu Feb 29 11:19:52 ICT 2024 : Certificate generation failed. Please contact Avaya support team
Thu Feb 29 11:19:52 ICT 2024 : Restoring original certificates
Thu Feb 29 11:19:52 ICT 2024 : Utility execution failed
Thu Feb 29 11:19:52 ICT 2024 : Changing execution modes of keystore files
Thu Feb 29 11:19:52 ICT 2024 : Changing ownership of postgres certificates to postgres user
Thu Feb 29 11:19:52 ICT 2024 : Restarting System Manager database service
Thu Feb 29 11:19:55 ICT 2024 : System Manager database certificate updated successfully

Anyone has the same expericence? And How to fix it?
And is there any command to manual change the SMGR's time and date to before the expired date?

 

[hcmbmt]

This system was installed longtime ago. Maybe the CA is expired. Cannot login to SMGR web admin, how can renew the CA for SMGR?

 

[Spookytooth]

You may be SOL. We tried it on two 6.3 systems recently and it failed. Avaya will not work on it and there is no v3 of the utility coming. The only user info we were able to get was from running pcap on sm100. We let it run for a while and eventually we got most of the users recorded by using their registration keep alives. Good luck you will need it!

 

[hcmbmt]

The Utility v2 only works if CA cert still valid. I tried to change the time of System Platform but also cannot fix it.
This system is Midsize Enterprise, after CA cert is expired, the SM is crashed then I have to reinstall it.

 

[bignose21]

Do you have any third part certs installed? could try the "sh CertificateRenewalUtility_v2.bin -FORCE

 

[kyle555]

There's a final patch for 6.3 SMGR that has a CA renewal utility built in. It's for SMGR 6.3.22

https://support.avaya.com/public/index?page=content&id=SOLN371062&actp=LIST_RECENT

 

[Craft01]

Hi all,

I have the same problem !!

@hcmbmt, may I know if you have a solution?

 

[hcmbmt]

@Craft01:
If 10 years CA is expired , you cannot use "CertificateRenewalUtility" to renew the cert. There are two options to fix it:
1. Reinstall SMGR.
2. Try to change the time of SMGR back to the time before the expired date to login to SMGR Webadmin. Then follow attached file to renew CA cert.
(How to change date: Login to SMGR CLI with root account - command: "date -s YYYYMMDD" - pls verify it - Centos CLI)

For my case, The System Platform controls the time of whole system. I changed the time of System Platform but it doesn't affect to SMGR and others. Then no choice I had to re-install all.

 

[Craft01]

@hcmbmt
Thank you for your quick interaction :, below more detail and accurately on the situation


1- This system is Midsize Enterprise, was installed in 2013,

2- For sure CA:
SPDOM CERT show 21/03/2013 to 19/03/2023
SMGR CERT show 15/08/2013 to 16/08/2023

3- I cannot login to SMGR web admin. I get an error msg:

""""""
The Web Server X509 certificate is expired.
This may be because:

The server clock has been adjusted so that it is no longer within the range of the certificate issuance and expiration dates.
The certificate has expired.

Click the following link to access the Local Login page. You will be required to log in using the credentials of a local administrator account.

/local-login
""""""""""""""""""
4- I changed the time of System Platform it Affect to SMGR and others. but I get a new error msg:

""""" /pages/Welcome.xhtml @70,67 value="": Error reading 'model' on type com.avaya.mgmt.console.framework.bean.ClientNavigationTree_$$_javassist_seam_6
Some internal error has occurred in the service. Please contact the support team."""""""""

5- I cannot Reinstall SMGR because I use Embedded Avaya WebLM Server for license, if I'll reinstall it I'll server Id will change and I'll lose licenses

I'm not using session manager, but CM get license from weblm and System Administration is Blocked

 

[hcmbmt]

@Craft01
For my site, after changing the time, I still couldn't access SMGR and error showed "The Web Server X509 certificate is expired".
Utility Webadmin couldn't access, SM crashed. Then I had to reinstall all.
If you re-install on same server, the host ID is the same, you can re-use the lic file. Or you can ask Avaya support to re-host the customer profile to your Avaya account then download the lic file from plds.

 

[bignose21]

Had similar at the weekend the SMGR root expired, this also ment that the id certs on smgr and sm expired at same time.

I logged into domain 0 of system platform as root
Stopped ntpd service and set time using date command to previous day
Accessed smgr with xm console
Ran the createCA.bin
Ran the CertificateRenewalUtility_v2.bin
Exit back to SP set time with date command
Enabled the ntpd service
Logged in to SMGR web, removed the session managers from replication
Set enrollment password in smgr
Accessed SMs and initTM -f

 

[Craft01]

HI bignose21,

this is not smgr 6.2?!! may be 6.3.22???

 

[hcmbmt]

@Craft01
I found this article in Avaya website. It's similar to yours:

https://support.avaya.com/ext/index?page=content&id=SOLN256442&viewlocale=en_US

Problem Clarification
Java Certification Expired

Cause
Certificates Expired in system manager
As it can be checked:

[root@hqs0591 ~]# openssl x509 -text -in /var/lib/pgsql/data/server.crt | grep Not
Not Before: Aug 14 07:51:32 2013 GMT
Not After : Aug 14 07:51:32 2015 GMT

This is for the apache:
openssl x509 -text -in $JBOSS_HOME/server/avmgmt/conf/tm/keystore/apache_load_balancer.pem | grep Not
Not Before: Aug 14 07:51:32 2013 GMT
Not After : Aug 14 07:51:32 2015 GMT
When the date has expired, the postgres cert needs to be renewed

Solution
Both methods are service effecting at the SMGR web :
Renew the ceritifates
If the postgres only expired, you can renew it without the tool:
[root@smgr ~]# /opt/Avaya/Postgres/9.1.3/utils/securePostgres.sh
After running above command, Restart jboss
[root@smgr ~]#service jboss restart
Test webconsole after 10 - 15 minutes.

 

[bignose21]

Yes 6.3.17 thought this was for a 6.3

 

[Craft01]

@ bignose21, LUCKY YOU
There is a utility which is included in release SMGR 6.3.9 onwards called createCA.bin which allows you to create a root CA quickly and easily in a 1 step process.

 

[bignose21]

Yep saved me going through the manual steps which I found would be possible in a small window.

If I didn't have the version beyond 6.3.9 then I found that if you ran CertificateRenewalUtility_v2.bin when it gets to the end bit where it is waiting for a script to run and counting up to 10 with 2 min delays (once it reaches 10 it will fail and revert the certs) there is a point that the SMGR web starts responding and could be accessed, in that window you could create a new Root CA, rename the old one, rename the new one to the default and assign it where needed (there is a doc on Avaya for that, all the steps done by the createCA.bin).

This is a link for all the manual steps but needs to be done from the SMGR web hence the above bit.

https://support.avaya.com/public/index?page=content&id=PRCS100850&viewlocale=en_US

 

[Craft01]

To login to SMGR Webadmin I have to change the time of SMGR back to 1 DAY before the expired date (I changed the time of System Platform)
Then following attached file (SMGR - New CA 6.1-6.2 Rev 2.docx) to renew CA cert. Set the date of the SMGR back

for now : it's ok when I go to

https://smgrIP@/network-login/

but still have error when I try

https://smgrIP@/smgr

""The Web Server X509 certificate is expired.

This may be because:

The server clock has been adjusted so that it is no longer within the range of the certificate issuance and expiration dates.
The certificate has expired.

Click the following link to access the Local Login page. You will be required to log in using the credentials of a local administrator account.

/local-login""

 

[serg0]

Hi. I have smgr 6.3. Before the end date root certificate, updated them according to this instruction (SMGR - New CA 6.1-6.2 rev 2.docx). And then I updated the identity certificates. But after the end date of the root certificate, smgr shows such a screen. What can be done?
"The Web Server X509 certificate is expired.
This may be because:
The server clock has been adjusted so that it is no longer within the range of the certificate issuance and expiration dates.
The certificate has expired.
Click the following link to access the Local Login page. You will be required to log in using the credentials of a local administrator account.
/local-login""
Нow can I log in using the link /local-login, the admin is not working? tell me who knows, please.

 

[hcmbmt]

Check the certificate expiry date of SMGR by:
- Login to System Manager command line interface as root user and run the following command:
$> openssl x509 -text -in /var/lib/pgsql/data/server.crt|grep "Not After"

If it's expired, try adjusting the system time to see if you can access the SMGR.

 

[serg0]

the certificate is valid until May 2026, and the web shows an error. what else can influence this?