Voicemail Pro V9.0.3 cannot send voicemail-to-email TLS/SSL issue? 2

[adc110]

I have a client Voicemail Pro system (yes still running on Server 2008 R2) that has been running for past 2 years on 9.0.3. We regularly use Voicemail-to-Email for a majority of users. Last Friday things took a turn and can no longer send voicemail to email. The phone system (V2 on same version) is sending alarms/notifications without issues so I focused completely on VM Pro. Looking through these logs I see that Voicemail Pro (once it establishes a connection to the mail server and after the EHLO command) issues the "STARTTLS" command. I think this is where things go down hill. If I use Putty and telnet to mail server and issue a STARTTLS command it fails as well. Originally we were on port 25 but modified to be 587 still with no luck. Would anyone have an insight on what might have gone wrong? I am not sure what TLS version is supported on 9.0.3. I also tried to find information on disabling STARTTLS in an effort to test (which even though is not optimal I just want to know if this is the cause). Here is a copy of the repeated SMTP Logs from Voicemail Pro which gets generated on every email attempt: Any insight or ideas would be greatly appreciated!



01/05/2019 12:07:18.591 - Attempting to connect to host mail.xyz.com on port 587, local bound address is (null)
01/05/2019 12:07:18.694 - Successfully connected to host mail.xyz.com on port 587
01/05/2019 12:07:18.748 - < 220 mail.xyz.com Microsoft ESMTP MAIL Service ready at Wed, 1 May 2019 10:07:17 -0600
01/05/2019 12:07:18.802 - ######## START OF CHUNK ######
01/05/2019 12:07:18.855 - > EHLO XYZ-COM
01/05/2019 12:07:18.908 - ######## END OF CHUNK ######
01/05/2019 12:07:18.960 - < 250-mail.xyz.com Hello [10.11.14.33]
01/05/2019 12:07:18.960 - 250-SIZE 71925760
01/05/2019 12:07:18.960 - 250-PIPELINING
01/05/2019 12:07:18.960 - 250-DSN
01/05/2019 12:07:18.960 - 250-ENHANCEDSTATUSCODES
01/05/2019 12:07:18.960 - 250-STARTTLS
01/05/2019 12:07:18.960 - 250-AUTH NTLM
01/05/2019 12:07:18.960 - 250-8BITMIME
01/05/2019 12:07:18.960 - 250-BINARYMIME
01/05/2019 12:07:18.960 - 250 CHUNKING
01/05/2019 12:07:19.077 - ######## START OF CHUNK ######
01/05/2019 12:07:19.130 - > STARTTLS
01/05/2019 12:07:19.232 - ######## END OF CHUNK ######
01/05/2019 12:07:19.285 - < 220 2.0.0 SMTP server ready
01/05/2019 12:07:19.386 - ######## START OF CHUNK ######
01/05/2019 12:07:19.439 - > EHLO XYZ-COM
01/05/2019 12:07:19.492 - ######## END OF CHUNK ######
01/05/2019 12:07:19.544 - CSMTPConnection::ConnectESMTP: An unexpected error occurred while sending the EHLO command error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure
01/05/2019 12:07:19.597 - ######## START OF CHUNK ######
01/05/2019 12:07:19.649 - > HELO VBT-VMPRO
01/05/2019 12:07:19.702 - ######## END OF CHUNK ######
01/05/2019 12:07:19.754 - CSMTPConnection::ConnectSMTP: An unexpected error occurred while sending the HELO command error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure
01/05/2019 12:07:19.807 - ######## START OF CHUNK ######
01/05/2019 12:07:19.860 - > QUIT
01/05/2019 12:07:19.912 - ######## END OF CHUNK ######
01/05/2019 12:07:19.965 - CSMTPConnection:isconnect: Failed in call to send QUIT command error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure
01/05/2019 12:07:20.017 - <
01/05/2019 12:07:20.070 - CSMTPConnection:isconnect: An unexpected QUIT response was received
01/05/2019 12:07:20.172 - CSMTPConnection::Connect: An unexpected HELO/EHLO response was received
01/05/2019 12:07:20.275 - CSMTPConnection:isconnect: Already disconnected from SMTP server, doing nothing

 

[derfloh]

Does the VM server trust the mail server's certificate?

Need some help with IP Office?

https://www.fwilke.com/home

 

[adc110]

I am not certain of this. Do you know which store I would look into for this?

 

[derfloh]

TBH no. Probably in the Windows certificate management. Look for trusted root CAs.

Need some help with IP Office?

https://www.fwilke.com/home

 

[adc110]

I actually don't see one specifically for that server. I think I will attempt to find an SMTP tool that is capable of issuing TLS/SSL commands to see if I can further narrow down the error. I already have SMTP Diag Tool but it doesn't allow the option of issuing TLS

 

[derfloh]

I had similar issues with Application Server. From one version Exchange servers tried to force TLS even if you connect through port 25 that is usually used for unencrypted connections.

You have to find out what certificate is used on the server side and who generated the server certificate. You have to ensure that you trust the certificate authority that generated the certificate and that you connect through an IP is FQDN that matches the server's certificate values (CN - common name or SAN - subject alternative name).

Need some help with IP Office?

https://www.fwilke.com/home

 

[adc110]

Thanks for all the posts! I am quite certain it is Encryption/TLS related. I wish I could find a simply TLS tool to prove this (rather than raw SMTP logs/Telnet tests). When I use Putty to telnet over on port 587 I do get connected to the Exchange server but as soon as I issue the STARTTLS I lose my connection. It all makes sense and still waiting to hear back from their internal IT on this. Thanks again for all the insight!

 

[adc110]

Derfloh - I did confirm that within voicemail pro I am using the CN that is issued to ("mail.xyz.com")and I am using this CN within the Voicemail Pro configuration as well (I am not using hostname or ip address) so I did confirm this!

 

[adc110]

Running voicemail Pro version 9.0.3. Does anyone know what version of TLS it is capable of? I confirmed that the company disabled all protocol versions except TLS 1.2 on Friday so this is most likely the issue. Not certain what version VM Pro supports?

 

[AACon]

The IP Office platform supports TLS v1.0, v1.1 and v1.2. All TLS interfaces start with TLS v1.2 but can allow negotiation down to v1.1 or v1.0 for compatibility. There are IP Office, Voicemail Pro, Web Manager and one-X Portal admin settings for 'Minimum TLS version' that enforce v1.2. Note that some Avaya clients do not support v1.2 at present.

-Austin
I used to be an ACE. Now I'm just an Arse.

 

[adc110]

AACon - would you happen to know where Voicemail Pro TLS modifications are made? (To enforce a specific version) The IT team advised that TLS 1.2 was enforced last Friday within their Exchange environment so at least I now have proof that something had changed. What baffles me is that they say this however the IP Office itself does not have TLS checked/enabled and is sending email notifications absolutely fine. I am thinking that because Voicemail Pro has been attempting TLS all these years and that lower versions were acceptable it might be the issue. Unfortunately the VM Pro logs do not prove what version of TLS is attempting to be negotiated. So, would anyone know where these settings can be made?

 

[t3rm3y]

AACon - would you happen to know where Voicemail Pro TLS modifications are made? (To enforce a specific version) The IT team advised that TLS 1.2 was enforced last Friday within their Exchange environment so at least I now have proof that something had changed. What baffles me is that they say this however the IP Office itself does not have TLS checked/enabled and is sending email notifications absolutely fine. I am thinking that because Voicemail Pro has been attempting TLS all these years and that lower versions were acceptable it might be the issue. Unfortunately the VM Pro logs do not prove what version of TLS is attempting to be negotiated. So, would anyone know where these settings can be made?

Did you find out how to set / change or check this?
We have a customer on 9.0 and they are migrating to office365, and their IT provided a SMTP auth client report which shows voicemail connecting only on 1.0 not on 1.1 or 1.2

 

[Westi]

I think this is the question of the day. As of which release of VmPro is the TLS 1.2 actually supported on VmPro.

I found this in the manual but that only pertains to Server Edition

· TLS Level Control
For secure operation using TLS connections, you can now select whether only TLS 1.2 or higher should be used.
This option is set through the voicemail server preferences using IP Office Web Manager. This option applies to
Linux based servers only.

We had 2 clls today already that customers with O365 got letters from Microsoft stating that the voicemail sends out with TLS 1.0 and that they will no longer allow that as of sometime soon.

All these customers are on 9.1 (.3 and .4 I think) and if we can simply upgrade them to R11 and be done then we make a sales team very happy but I cannot find information that confirms that.

Joe W.

FHandw, ACSS (SME)


"This is the end of the world, make sure to buy your T-shirt before it is too late"
Original expression of my daughter

 

[t3rm3y]

Hopefully there is a fix as our customer is a care home with multiple sites and old handsets so upgrading to r11 is not such a simple process.

 

[derfloh]

Then use a SMTP relay that receives the mails from VMPro and forwards it to the upstream SMTP server with the needed TLS version.

Need some help with IP Office?

https://www.fwilke.com/home

CLI based cale blocking:

https://blog.fwilke.com/ip-office-block-calls-from-specific-cli/

SCN fallback over PSTN:

https://blog.fwilke.com/ip-office-scn-fallback/

 

[aarondyck]

I've had issues with voicemail authentication in the past - my solution was always to bypass the customer's SMTP server! I have a web hosting package that allows unlimited e-mail traffic. Set up a customer@domain.com e-mail and configure the IP Office to send using my mail server. Problem solved.

Of course, not every customer is happy with that. An SMTP Relay will work, as suggested by derfloh. I always tell the customer that they can provide an SMTP Relay because I don't want to have to support it down the road. They configure the relay per their requirements and I simply send the e-mail to the relay point and from that point it's the customer's problem.

http://www.adventuresinvoip.ca/

 

[adc110]

So, I upgraded customer system from version 9.0 to version 11.0.4 over the weekend and Voicemail-to-email is still not functional. Same errors, so I threw wireshark on the VM Pro server and observed and as you will see from the attachment - it is only used TLS 1.0. And, since customer Exchange server is forced to use 1.2 it will not negotiate and fails with all the same messages as I previously posted. I have searched high and low for "accurate" information from Avaya to determine how to get around this or force VM Pro to use 1.2 (or at least auto-negotiate which it is not). Running a mail relay on this particular customer's system will not fly with them as they take security very seriously and sticking a mail relay is not going to be an option. The thing that baffles me the most is the fact that the old ContactStore server, the IP Office itself, and even CCR before I just displaced it after upgrading over the weekend still can work with customer's exchange servers sending messages all the time without issue and therefor negotiating correct TLS version. You would think VM Pro would follow suite??

Thank you for all the posts above - I felt like the crowd needs to know that latest version of VM Pro does "NOT" solve the problem unless I am missing something. All of the documentation points to Serverd Edition or UCM wherein (apparently) you can set TLS versions but definitely not on a Windows server based VM Pro installation of the software.

 

[adc110]

 

[adc110]

So it was confirmed through ATAC that the support of TLS Version 1.2 is "ONLY" on their Linux version and not Windows version. This is quite unfortunate to know that development has not included this on both platforms. That being said, thank you all for your comments/suggestions. I did end up downloading/installed hMail on the new server and forwarding all emails to hMail and then use hMail as relay to internal Exchange and working perfectly (hMail does support TLS 1.2). Certainly not the best solution but as most companies are not excited about installing a relay in their network but hMail does allow locking itself down to only authorized connections (in this case Voicemail Pro and hMail reside on same server to I restricted access to the relay from itself). Oh well . . . . .

Thank you derfloh & aarondyck. Relays have worked nice in the past but I was hoping to avoid it at all costs but not going to happen in this case . . .

 

[critchey]

Thank you for posting the resolution. Someone will eventually run into this issue I am sure.

The truth is just an excuse for lack of imagination.