playback of voicemail through browser and/or OneX Mobile doesn't work 3

[qtelcom]

Hello, Ip500v2 9.1.5 and Linux avaya App Server 9.1.5

Linux avaya app server is the voicemail pro and onex portal server. I can not play the voicemail back via the browser or OneX mobile app. No error messages. Can log in fine, the voicemail provider is green in OneX Portal health components. I see the play button when I log in to OneX portal, looking at voicemails, it says via browser, but it doesn't do anything when i select the voicemail and click play. This also doesn't play within the OneX Mobile app, I can log in fine, see the voicemail but cannot play it.

I have this working on our app server in house so I know it works. What could be the issue with this? I thought this just worked out of the box when using the linux app server? Is there more to this to make it work?

 

[kyle555]

I've got a ticket open about this for months and they wrote a patch which might/might not be working!

So, do you have a "public IP" defined in "network topology" for a SBC for either remote workers or SIP trunks?

 

[qtelcom]

Don't even care about outside access yet. I'm just trying to get this working on the internal side. Have a DNS record that points to the internal app server ip. So local user and local onex mobile app user cannot playback the voicemail. What makes this work? Does it look up a URL to playback the VM or what exactly does hitting the play button do?

 

[qtelcom]

but to answer to your question, I do have public IP set on the LAN1 topology. My remote OneX mobile users can connect fine, gets available and presence is green as well. Voicemail playback doesn't work for them either.

 

[kyle555]

The reason I asked about that public IP is because it was what was tripping me up due to a software bug.

I've been working on this for a couple of months. Now, bear in mind I'm working with a 9.1.4 all-in-one server edition, but here goes...

Playing voicemail through one-x portal works in 2 ways.
1. The Portal fetches the voicemail wav via https from VMPro and delivers it to mobile clients.
2. The Portal provides the voicemail URL to the browser or Outlook Plugin for the PC to fetch the voicemail wav.

I'd noticed a couple of problems - namely, that the public IP in that network topology field I was mentioning seems to overwrite where One-X Portal looks for voicemails. Neither PCs in the LAN nor the Voicemail Server can access vmpro on the public IP, hence playing the message fails.

Now, what's even more goofy about it is that if you change that "public IP", you need to restart IPO, but even if i restart Portal after, the change still won't take effect until I totally reboot the OS of the server. Once I started changing that public IP to "1.2.3.4" or "9.9.9.9" and rebooting the OS, I could see Portal looking for voicemails at "

https://1.2.3.4/users/etc/whatever.wav".

You can check that out too if you make test user "Sally Applesauce", leave her a message and try to play it in the browser. Just make sure to turn up the logging in the portal to "all" first.

Log in to your portal as root and:

grep -r "Sally Applesauce" /opt/Avaya/oneXportal/9.1.5.releasestring/apache-tomcat-logs|grep "VM url"


You'll see what URL Portal is providing for that voicemail. And I'll bet ya that you can paste it in your browser, and just change the IP for the VMPro IP and it'll play to prove VMPro is actually delivering it.

You can also wireshark from the PC with the browser you're trying to play from and filter on port 5443 (that's the web voicemail port) and see what your browser was instructed to do to fetch the voicemail message.

Because portal fetches messages from VMPro and delivers them to mobile clients, you can easily hack around that with an iptables rule on portal that redirects all outbound traffic on 5443 to localhost. It isn't exactly an elegant solution, but I tried it and it did work.

If you've got the same problem I do, check in with Avaya and mention portal build 9.1.407.37 that they made for my issue in 9.1.4. They confirmed to me it exists in 9.1.5 and asked if I wanted the fix put in that release to. I might just wait for it to get in the main 9.1.6 before I patch though.

 

[qtelcom]

did everything you mentioned. Turned logging to all in OneX Portal and left a message for the user "test"
tried to play from the onex portal browser interface logged in as "test"
Nothing played, no errors, nothing...

Ran this command via putty as root.
grep -r "test" /opt/Avaya/oneXportal/9.1.500_9/apache-tomcat/logs|grep "VM url"
Saw a bunch of stuff, but saw this string of interest.

https://10.101.87.90:5443/VoicemailAccounts/test/dd991f659519634783549d45695aac88.wav

I plugged that into my broswer and it had a security warning so I added it as an exception and it downloaded the wav file and it played fine.
Went back to the browser for OneX portal logged in as "test", and now this user can play the voicemails from the browser via OneX Portal. I'm guessing this is a security certificate issue? but just for voicemail playback?!?! I already permanently stored the exception in firefox when I first logged in to OneX Portal as "test".

Still can't playback via onex mobile app internally.
Any ideas?

 

[kyle555]

I suppose it could entirely be possible that its a certificate issue if the web server hosting VMPro and the WAVs isn't the same as the web server hosting Portal.

Maybe you've got a bug where the most permissive security settings - like ignoring certificate errors are preventing Portal from fetching the message for your mobile device. But, if that 10.101 IP isn't the one in your Public IP in Network Topology, then you wouldn't have the same problem that I did.

Maybe VMPros http logs might help debug what happens when you try to play from the mobile client. And you can look deeper in those izwi inyama inkama whatever logs to see what the process is for Portal to get those wav files to deliver to the mobile clients. Maybe you'll see an SSL error or handshake failure or something.

 

[qtelcom]

think I might just reinstall the app server.... I did a backup/restore of the current onex portal windows version to the linux one, but I want a vanilla onexportal from scratch now. I see you can uninstall onexportal from the web management, can you easily reinstall it? or must I reload the entire app server. Not sure if this is documented on how to restore OneXportal to factory defaults. Anyone know?

 

[qtelcom]

that 10.101 address is not my public ip defined in LAN1 network topology, so it isn't the same bug I guess.

 

[kyle555]

I have no idea re: reinstalling one particular app of the app server.

Regarding the whole "not playing from mobile", I'd still troubleshoot it to see what you get in the logs and stuff just for comparison's sake if you still have the big on a fresh install.

I can tell you for certain that a fresh VMWare install of all-in-one server edition has issues, so don't be surprised if you still have a few!

I was bugging Avaya to use the VMPro FQDN for the VMUrl so it could be resolved inside or outside the network based on DNS and not tell Portal users browsers outside the network to fetch voicemail from a 10.x address, but that's another gripe altogether.

 

[Pepp77]

In One-X do you have the voicemail provider set as 127.0.0.1 or the actual IP address of the voicemail server. If the former try the latter. Has fixed this issue for me on more than one occasion.

| ACSS SME |

 

[qtelcom]

I did double check, in One_X the voicemail provider is not 127.0.0.1, it's 10.101.x.x which is the IP of the App server.

Playback of the voicemail through the browser is working, seems to very browser dependent though.

I still cannot get it to play via the one-x mobile app, we've been testing on the internal network via wifi with the mobile app to eliminate the external router/firewall. (it doesn't work outside the network either) Specifically speaking, when I get a new voicemail on my test user, I can see it in onex mobile via the events tab, and click the inbox, I see my voicemail with a blue dot, once I click on it, I see an icon spinning right below it like it's trying to load it, but it never does. The rest of the mobile app functions fine, presence, VoIP calling, conferencing, etc...

My understanding from my testing (and I don't know how it really works behind the scenes), it wants to download that voicemail wav right into the oneX mobile app, but only that first time you click on a new message, and then it must have a copy of it stored on your mobile device within the app. (on my working system in house, it appears to only need to load it the first time, and subsequent tries it already has it loaded, as there is absolutely no delay when you listen to the same voicemail the 2nd or 3rd time, only the first time there is slight delay as it grabs the voicemail) btw, on the non working system, the message is marked as old even though it never played it to me, so the very act of clicking on the voicemail is telling the voicemail I'm listening to it and to mark it as old so I guess that part of the communication is working.

I looked at OneX Portal/voicemail logs on the app server from the time I first click on the new voicemail on the mobile app, to try and see if anything jumps out at me, but I don't know what I'm really looking for and those various logs contain tons of information.

Next steps for me is to completely reload the App server, and not restore the backup of their voicemail pro or the oneX portal DB. So basically a vanilla Linux app server with their current IP500v2 config and see what happens. (They are coming from a Windows VMpro/OneX server to Linux App Server, I had backed up the windows VMpro and OneX DB & restored them on the Linux App Server)

 

[kyle555]

I just thought of something.

Maybe the widget app within the portal browser refused to play the message because you'd never connected to

https://yourIPO:5443

for voicemail.

Like when you accepted the certificate when you connected to Portal on 8443 or 9443, and you accepted the security exception of the self-signed certificate most likely in the format "IPO.MACADDRESS".
Except when you popped that VM URL in your browser on port 5443 and finally accepted the cert, the widget in the portal application would play voicemail.

Not really related to my issue, because my customer is using 3rd party certs.

Avaya released article SOLN276160 to describe how to do 3rd party certs with IPO for this purpose. Maybe you could lab it out and try using a godaddy or verisign or your customer's CA on the IPO and see if it changes anything.

That, or "tshark -i eth0 port 5443 -w /tmp/capture.pcap" as root on the app server would fire up a wireshark in the command line and listen for traffic for web voicemail on port 5443 and write the output to /tmp/capture.pcap for you to look at after. Presumably the mobile client is asking the portal application to fetch the .wav from VMPro and it would at least let you see you're getting as far as getting the message out of VMPro and into Portal and just failing to deliver it to the mobile client.

 

[OhashiNoel]

What if you use Microsoft Exchange Server and Outlook?

 

[qtelcom]

Late to update this, but after 3 weeks of the case being open, I got a patch that had already been released for this issue. I think it's the one you mentioned in the beginning Kyle555. I put that in and it fixed it right away, it was a 16kb patch, (yes 16 kilobytes). 3 weeks of case notes, back and forth traces and logs on Avaya's support site, just to get an already existing patch... sigh... Could've been a 10 minute fix, but oh no Avaya... turned into 3 weeks long, and multiple trips out to customer site. I've never been impressed with their support. Several times my case would go into neverland and would be days before the support tech would update the ticket. Had to press the escalate button, which basically must pop up a big red flashing red alert light to the manager since I got a call within 10 minutes of pressing that button. I would complain to my CAM but our CAM's change every 3-6 months lately for us...

 

[derfloh]

I'm curious how they want to be able to handle the support calls if everyone will have IPOSS when R10 is out...

 

[kyle555]

Lucky you. Still broke for me as of 9.1.6!

 

[JRTrevino]

@ QTELECOM, what was the specific release version they gave you? I have the same error occurring with a client on the latest GA Release 9.1.6. Thanks.