Avaya IPO Voicemail Pro losing connection random HELP!!!

[PTG]

We have an issue we have been dealing with on and off for about 4 months. We have a 5 site system using voicemail pro, installed about 3 years ago. The system is running 6.0(18) firmware. The issue is the voicemail connection is lost and we have to restart the server to get it back each time. When you look at the "services" it is always still "Running" but its not linked. Even if you start and stop the services, it wont resync until the server is rebooted. It was on a stand alone PC for about the first 18 months when the issue started. We were running Vista on server. We did a fresh install using XP on the same machine and issue still there. So then we moved the voicemail to a new machine (a virtual server)at a different SITE even, fresh install, etc. and it was fine for about 3 weeks, then it started acting up again. We have checked everything on the server we can think of. I will look in system status and the error I see is just "voicemail pro connection failure". I will sometimes see an error for "ports unavailable" or stuff like this but not always. We have checked the connection between this site and the voicemail over the MPLS and there is never a loss of connection. It happens at random times anywhere from like 1pm in the afternoon to 3 am in the morning. Sometimes during heavy call traffic, sometimes no traffic. It will do this 10 times in one day for a couple days, and then for a few days nothing. Then we got no issues for all of last week! The license is valid in the IPO. We have worked with their IT guy very closely and we are all at our wits end on this. Our programmer has done about 200 IPO installs including several dozen VM pro, and never seen this issue. So the only thing in common that has not been changed from the first server to now is the specific control unit the license is housed on, the voicemail license, the IP address it has, and the MPLS network they have. The only thing we can think of it the firmware? maybe update to latest? or switch out the control unit just to rule that out? or get the license swapped with Avaya? any other ideas anyone??????

 

[intrigrant]

I have seen this once, it was a salesman who setup a fixed ip address on his laptop and yes, it was the ip address of the IP Office.
There can be a million reasons why it happens it seems, the only thing you can do is monitor as much as you can with all available tools like sysMon, DebugView, Wireshark etc.

A simple mind delivers great solutions

 

[Telecomboy]

Might be simple, but make sure the CPU LAN port is not set to be able to hibernate.

 

[PTG]

We have checked all the setting with the server, network, etc. we have been working on this for months. So yesterday we ran the Avaya "debug" program so we could reference that after the voicemail quit again, checking the time it stopped, and see what we saw in the dbug log. But when we went to check it, it was just missing about 3 minutes of time there. To clarify, according to system status the IPO lost connection with the VM pro at 3:08am. When we look at the dbug log, it is missing times from 3:07 to 3:11, then starts on again at that time. So its like it didnt record anything because the connection was lost. This is on an MPLS network and we have no evidence of loss in connection between sites. The VM sever is at a different site than the control unit that the VM pro license has. We are at our wits end on this. Any more help greatly appreciated...

 

[amriddle01]

The issue will be the link and/or the network, I am 99.99% sure. Finding it isn't going to be easy as these types of things never are and fixing it ...probably worse, but take your focus off the server and software and concentrate on the link/network..... that's what I would do


Avaya Implementation Qualified Professional Specialist Technical Engineer (AIQPSTE)

 

[markusoreallyus]

Mirror the interfaces the IPO and VMP use. Run wireshark on two computers connected to the mirror ports with rolling captures. IN theory you will be able to match up the normal operation packet for packet (for a small timeframe) and then prove the IPO is still sending packets to the VMP that the VMP never receives. I always wonder if there is a packet rate limiter that audio conversations can quickly overwhelm, kind of like limiting broadcast storms by dropping broadcast packets.

 

[budbyrd]

I have seen this at several sites and it has always been a duplicate IP address. More often than not it is some piece of equipment that wont respond to a ping.

 

[HyTech]

Try File--> Advanced--> Erase Security Settings on IP Office, then resart VM Pro Service......It has worked for me quite a few times.

 

[amriddle01]

Good point, setting a ping away and unpatching the pc briefly will prove this, then check your arp cache to find the MAC address and it may reveal what device/manufacturer to look for


Avaya Implementation Qualified Professional Specialist Technical Engineer (AIQPSTE)

 

[cruzades]

@PTG


Are you sure that the PC where the VM Pro installed is standing by 24/7?

I'm thinking of its windows' power configuration, it should be stay-on all the time, e.g. Hibernating, Power-off, HDD off, etc.

 

[PTG]

Thanks for the input guys. To answer the last question, yes we are sure that its not a PC issue as far as hibernating or a power setting. Sometime the VM pro will stop ten times in one day, during heavy call volume, but then other times stop in the middle of the night or like 3am during no call traffic. Then it may go 4 days just fine.
As far as the other comments made, I will take the server down while I send a ping to see if I get anything on the network to respond. This way I can rule out the IP conflict that you guys suggested.
I will also try the fix that "Hytech" suggested to try.
I too really believe that it is a network issue with the link between the server and the control unit. They currently are at different sites connected via the AT&T MPLS network, but when we had the two at the same site together we got the same problem. I tried to run a continuos ping from the server to the control unit to see if we ever had a connection "hiccup" or anything about the time we get the disconnect link error. No luck yet though....

 

[tlpeter]

What about swapping the server for a different one?
This already takes a week and that would be the first thing i would try.


BAZINGA!

I'm not insane, my mother had me tested!

 

[PTG]

Tlpeter,

We are on our second server already. Entirely new operating system, fresh VM pro download, etc. same issues as before..

 

[tlpeter]

OK, missed that part.
Then take it off the customer network and put in your own temp switch to proof network related issues.


BAZINGA!

I'm not insane, my mother had me tested!

 

[PTG]

Taking it off the customer network and putting on its own switch could help prove stuff, but they have 7 sites and they all share the voicemail so they all need to have access to it. Maybe if we put it on its own vlan or something...?