Aura Stopped Working Suddenly 1

[lukesimmons101]

Our voicemail server hadbeen down for over a year when I reported, with the help of an Avaya technician we restored our CMM server to Aura 6.0.1-8.0 with SP4, updated the subscribers list to be current, and everything was great. Two months later it just doesn't work any more, pressing the message button on our phones or calling the extension just gives a busy signal. I looked in the web interface and the messaging starts and stops like normal, all LDAP services are up, so I'm at a loss. GEDI is R016x.00.1.510.1. Where should I look / check? What further information would you need to be able to assist? Thank you in advance!

 

[kyle555]

stat the sig and trunk group in CM for voicemail

 

[KenMeans]

Is this a CMM or A AAM system ? If you get a busy you need to look at the SIP trunking first. Your statement that you restored the CMM to Aura makes no sense.

Ken Means

http://www.tel-is.com

"I find that the harder I work, the more luck I seem to have."
- Thomas Jefferson (1743-1826)

 

[lukesimmons101]

Sorry for the mix-up, I don't know all the proper terminology for everything. As far as I know it's a CMM, that's what appears on the web interface. When we restored it was to fix a hard drive failure, we basically reinstalled from scratch to restore functionality. My apologies. Stat trunk 99 gives all 51 members of the msgserver trunk group in-service / idle and status signaling-group 99 is in-service as well.

US Navy IC1
2012 - Now
“Do what you can, with what you have, where you are.” —Theodore Roosevelt

 

[kyle555]

OK, so it looks like there's SIP talking.

The trunk group has a TAC. It's in the top right if you display trunk 99. If you list trace tac #099 or whatever it is, you can see a call try to go across.

Maybe check the switch link config in CMM as well and see if there's a channel monitor showing if they're up or not.

 

[lukesimmons101]

The trace didn't return any results, just shows starting up message and then nothing. I called the extension for our voicemail twice and no log entries. All 33 channels *On Hook with 0 calls today. What is a good setup for the "Dial Sequence" page? I currently have Dialing Sequence N, Account Code Length 0, and Account Code blank. That seems "off" to me.

US Navy IC1
2012 - Now
“Do what you can, with what you have, where you are.” —Theodore Roosevelt

 

[kyle555]

list trace station xxxx where your station is the one calling voicemail

 

[lukesimmons101]

dial 6444
denial event 1026: AUDIX Unavailable D1=0x9da1 D2=0x1525
dial 6444
reorder hunt-group 1 cid 0x1525
idle station 6573 cid 0x1525

So am I correct in saying that it's trying to contact the hunt group for AUDIX, but can't?

US Navy IC1
2012 - Now
“Do what you can, with what you have, where you are.” —Theodore Roosevelt

 

[kyle555]

Well, yeah. That's what that means!

Now, I'm just talking out loud here, but if I were called in to bring up a voicemail that was on its arse, I'd make sure it was working before I left, yknow what I mean?

Now, you say Avaya Aura 6.01-8.0. Do you mean release 6 or 8?

On R6, CMM was included with the CM image, so it was the same server, same IP, and if you disp sig 99 and list node-names all, you'd see CM and CMM have the same IP and the sip sig group was between ports 5060 and 6060 with CM using 5060 and CMM 6060.

At R7+ CMM is its own virtual machine, so it has its own IP. You can tell that pretty easily displaying the sig group and checking the IP of the far end node.

If you're saying you needed just a CMM restored for a bad disk, I'm almost inclined to think you're on R7+ otherwise you'd have had a CM server restored with that too and on R6, CMM was only available on CM simplex (1 server, not the 2 HA server pair), so if you had a server die on R6 with CMM, I'd reckon you'd have been without a CM too.

So, if it is on it's own separate server, it would need to be pointed to WebLM. You're probably using System Manager for that, but in the CM webpage in the menu at the top you can click Licensing and see where CM is pointed. It's pretty reasonable to assume CMM is pointed to the same place. Access WebLM and see if any CMM licenses are occupied. If not, then maybe buddy restored your backup and missed pointing it to the license server, so that could be a fairly good reason why CMM wouldn't answer.

I'm guessing with you being Navy, you might have a dedicated CMM server on R6 which is possible for a special Federal Markets offer and I believe it could also do H323 signaling where that was deprecated for the most part at R6.0 or 6.2.

Otherwise, did buddy fixing your system back up run through the Military Unique Deployment Guide? Cause there's special stuff in there around securing the system. On the surface it doesn't seem like a big deal cause a sig/trunk group in service means at least voicemail is answering with SIP or H323 responses, so you know someone's home as far as the software running is concerned.

So, what else can you tell us?

And, what's in your alarm log?
And, when you click Start Messaging, paste the tail end of what's at the bottom.

I've seen situations where it thinks it's "started" but the tail end there will show you something like LDAPFE is not responding... repeatedly and it's technically running but it's still broke.

 

[lukesimmons101]

We definitely have separate IPs and VMs for the CMs and the CMM, and we didn't have any down CMs when we replaced the hard drive, so it seems like we have the special federal markets setup you described. I'm unsure if the technician was following the Military Unique Deployment Guide, but I can reach out to him and find out. As for amplifying information, license status for the CMM is as follows;

No valid CMM license file installed on WebLM server
Checking application Messaging 6.0-1
Messaging license mode: ERROR

That doesn't make sense, the CMs have valid licenses and there's no reason we'd have different lengths of time on the license. I cannot access the WebLM server interface, I'm working to get those credentials currently.

Alarm log is empty of active alarms, a snapshot of resolved alarms since 7/15/19 are:

App Resource Location Alarm Code Alm lvl Ack Alarmed date Time Resolved date Time Resolved reason
VP LICENSE 2 MAJ N 04/19/20 08:53 05/01/20 09:49 RESTRT
VM SERVER ela-de 6 900 MAJ N 05/01/20 09:50 05/01/20 09:51 MAINT
VP VOICE_PORT 1 MAJ N 05/01/20 10:10 05/01/20 10:13 RESTRT
VP VOIP IP CA 7 MIN N 05/01/20 10:11 05/01/20 10:13 RESTRT

They're like that repeating all the way back to 08/10/19, and we lost voicemail at the end of August. As for the start messaging log, there's a line that says "Attempting to start LDAP server PASSED" and "End: start_vm: Voice Messaging is now Completely up" at the end with the date and time. I'm not the only one with access to this space, so someone could have inadvertently changed something that removed an association or path the system needed. Have you ever heard of that happening?

US Navy IC1
2012 - Now
“Do what you can, with what you have, where you are.” —Theodore Roosevelt

 

[kyle555]

If it's 6.0, then I'd reckon you're on System Platform, which has it's own WebLM server in the Console Domain.

So, you install System Platform and it gets 2 IPs, one for Domain0 that's where the virtual machines live and the first virtual machine is Cdom - the console domain - which is a fancy word for a web page management interface to manage the server from. Load up your CMM, set network interfaces, NTP, DNS, that sort of thing.

Presumably your CMM webpage would have a spot for licensing and you'd see where it's pointed - it could be pointed to itself or possibly to something more central. If you or your partner have access to PLDS, you can get and regenerate licenses. Typically they're against a physical mac address of a particular WebLM server. So, if you just changed a bad hard drive for example, your old license would be fine. If you changed the server, you'd check what the new HostID was in that new WebLM and regenerate your license on the fly in PLDS to get something to make the server work.

Otherwise, yeah, you got a license problem.