Connectivity alerts with VMPro 4.23

[SJCA53]

We have 10 locations across 5 states on IPO 4.2.14 using an ASCN to connect: 8 406 controllers and 2 412 controllers with 1 VMPro 4.23 voicemail server. We have a fairly robust MPLS WAN, with at least 1 T1 in each location, and QoS is set up to prioritize voice traffic over data. Since March, we've had random alerts from the remote offices stating that the 'Voicemail Server is down', then three seconds later (literally)'Voicemail Server is operational.'

It is always the same amount of time, and there is no discernable pattern to the alerts. We've looked at all of out network monitoring tools and cannot figure out why this is happening. Any ideas / suggestions on how to track this down? It mostly happens after hours, but we have had a few during the day, and it breaks connections to voicemail when this happens.

 

[NYSTECH]

What changed in March??? Upgrades to IP Office or VM system.. Any upgrades on network at main site?

ACS- IPOffice Implement

 

[SJCA53]

That's why it is so weird. We switched WAN providers at the end of January, we didn't do any IPO upgrades or VM upgrades until later in March, so there wasn't any change that we could point to with the LAN or WAN or equipment / version changes that might have caused this.

 

[amriddle01]

You don't know what changes your WAN providor has made since then though :-0 Have you asked them? What does monitor say when it goes down?

ACS - IP Office Implement

"I'm just off to Hartlepool to buy some exploding trousers

 

[kwing112000]

we had a site like this recently and we found a duplicate IP on the network.

Kevin Wing
ACS- Implement IP Office
ACA- Implement IP Office
Carousel Industries

 

[NYSTECH]

I am confused.. Sorry! After the Wan changes did you start having this issue?? or did issue start after you upgraded IP Office and VM?

ACS- IPOffice Implement

 

[SJCA53]

Duh. Sorry about that - I thought we upgraded later in March to 4.2.whatever, but it turns out we did do the 4.2 upgrade just prior to the alerts showing up. Hmm. The Avaya techs weren't finding any indications of issues when we sent them the traces. We've since upgrades to 4.2.14, but the alerts are still happening.

Anyone else having this?

 

[NYSTECH]

Where are you seeing these alerts? In SSA or some other minitoring application..

Did you also change out VM server/PC for upgrade?

ACS- IPOffice Implement

 

[SJCA53]

We are using the SMTP settings in IPO Mgr - System to send email alerts of different events to a group of IT staff.

The events it sends alerts for are: Application, Compact flash card, Expansion Module, Trunk, VCM, Generic, License, and Loop Back. Phone change is turned off because we do that a lot.

The alerts just say 'System alert' in the subject header, so I guess they fall under the 'Generic' heading?

We did not change out our VM server, but we did upgrade it along with everything else when we went to 4.2.14.

 

[NYSTECH]

And you say users complain about VM being down??

It definetly sounds like something to do with the LAN that it resides on..
As mentioned before maybe duplicate IPs or some sort of QOS issue... have you added any other equiptment/application to the Data network around same time of upgrade of PBX..

Since it happens at differnt times very hard to find out..

ACS- IPOffice Implement

 

[SJCA53]

The only complaints we've had are when this happens during the day and someone is connected to voicemail.

Each office has it's own LAN/subnet, and the controllers, servers, voicemail servers, printers, routers and firewalls all have static IPs that are not available to DHCP to be assigned. I'm not sure where the duplicate IPs could be happening that would cause an issue.

We didn't add anything new aside from the upgrade, and the randomness is making it tough to figure out.

 

[amriddle01]

To test, set a constant ping away to both the system and VM PC and power them down and see if the ping comes back up or continues before you power the kit back up, if it does you have a duplicate IP address, if so type arp -a and you will see the MAC address of the offending equipment

ACS - IP Office Implement

"I'm just off to Hartlepool to buy some exploding trousers

 

[SJCA53]

I can do that, but only during a maintenance window. I'll have to try that Sunday during the next window.

thx

 

[Telgian]

SJCA53, did you ever find out what the problem was for this? Did you find a duplicate IP? I have the exact same thing going on right now since we upgarded. Seems that the latest version is much more finicky than previous versions. I found that when I turned off our backup job to VM server the errors went away. We use DPM and Backup Exec for different backup schemes and when I disabled those jobs I didnt receive any errors for 2 days. We're trying to figure out why this is happening.

 

[SJCA53]

Interesting. We don't use BackupExec to back up that server because we didn't want to complicate things. We don't have duplicate IPs, and the number of instances of this has dropped dramatically in the last month. It's really crazy because it is so random. I do agree about the finicky part -we won't do anything to that server because of previous experience with really twitchy phone servers!