BCM450 callpilot gives immediate "Your message is too long" after greeting plays

[QueBall780]

Having an intermittent error in the voicemail system here.
Running BCM450 R6 SU 17 (Latest and greatest patch available)
Started happening a few days ago. Was running SU15 at the time, tried upgrading to SU17 as part of troubleshooting but it has not changed anything.


What is happening is that the callpilot subsystem is immediately telling the caller that the recording is too long right after the greeting has played then they get dumped back to the mailbox menu. Pretty much all callers at this point will hang up and call back to complain to the receptionist that it didn't work.

Internally just hitting feature 980 and tring to leave direct messages in the mailbox I get the same problem. So does not matter if the call was from an outside trunk or just an internal intercom call the only difference is instead of the recording that message is too long it shows the text "msg too long" on the display of my phone. With the menu on the softkeys for retry play and quit.

The problem will affect random mailboxes but once it starts it happens with every attempt to leave a message on that mailbox. I found that if I keep doing retry on that mailbox it will allow me to record a message like normal and then that mailbox starts working again. So I went through the entire list of mailboxes and hit retry anytime it came to a box that would complain the message was too long but the problem is returning as more mailboxes are getting tied up with this bug.

I checked for free space, the callpilot says there is 24861 minutes of recording space available and when I download the logs from the BCM and check the output file showing the disk free command it shows there is lots of space left in all the various linux partitions of the system so it doesn't seem to be running out of space.
MESSAGE USAGE REPORT Date: 2013/05/03
Minutes
-------
Current storage available 24861
Filesystem 1k-blocks Used Available Use% Mounted on
rootfs 4134932 941384 2983500 24% /
/dev/md5 132221 6754 118640 6% /nn/altroot
/dev/md6 15488744 1141576 13560388 8% /var
/dev/md7 33032228 96008 31258232 1% /var/shadow
/dev/md8 6198432 129456 5754104 3% /var/log
/dev/md9 5162828 379596 4520972 8% /nn/patches
/dev/md1 48052 10136 37916 22% /nn/images
/dev/md2 48084 0 48084 0% /mnt/hdd/sdb2
/dev/loop0 2023920 40 1921068 1% /var/secure/mp0

The unit has mirrored hard disks and the mirror status shows it's normal.

There are no errors in the alarms to help explain things

I tried editing the class of service that these mailboxes are using. But some of the mailboxes that have had problems were on different class of service, some were on COS 1 some were on COS 3 which are the two settings we use.

The mailboxes having problems have been users with 0 messages stored and users with a few older messages stored and one user who had dozens of old messages.

When the problem first started happening I tried to go into service manager and shut down and restart the voicemail service in element manager but it would not start up again. I had to restart the entire BCM to get it back again. It took a very long time to shut down the service and then it takes a long time to attempt to start it again and eventually comes back with error saying it could not be started. If I refresh the services screen it shows the service is running but doing Feature 985 or any of the voicemail feature codes shows inactive feature so it's dead.

Unfortunately I cannot make any sense of this one and I don't know what else to look at. Any hints?

 

[QueBall780]

Talked with Avaya.

Has been suggested it's likely either some kind of corrupt database in the call pilot voicemail system and I need to reset to factory default (Feature 982, reinstall code etc)
Told not to restore a backup as it could just reintroduce the problem.

Next thing to try is replace hard drive as it may be an odd symptom of bad sectors where the drive takes a long time to respond and the process times out or freezes.

Yay. Weekend plans out the window, going to be going through making notes on all the configuration settings, backing up all the greetings and making copies of any messages that are waiting for people and then blowing it all away.

Going to replace the hard drive too for good measure. Thinking I'll get a SSD drive as a replacement.

 

[QueBall780]

Ok,

So I did all this backing up of call pilot settings, company greetings, recorded names and even most of the voicemail messages via unified messaging feature.

Then I go to replace the hard drives just to cover the off chance that the old drives were a problem even though the RAID was fine and there were no errors.

Well, I take the original main drive, stick it on my USB to SATA adapter and the drive is struggling to even spin up. Making very bad noises. Times out over and over again, I could not get the thing to spin up at all.

Turns out I have rebooted the BCM many times in troubleshooting but not once have I cut power to it. If I had I'm sure the drive would have never spun up and at best the mirror drive would have booted and at worst some kind of post error would have blocked it from starting up until the drive in bay 1 was pulled.

I would say I am shocked and disappointed that the RAID and logs never showed the drive failure but in my experience it seems to be normal that nothing appears wrong with the drives until it's catastrophic failure time.

So I'm not going to reset the settings just yet. With an obviously failed drive on my hands it seems likely it was the cause of all problems. The mirror drive appears healthy at this time and I'm going to give it a day to prove if it's working or not. If there is some kind of further database corruption then I'll go ahead and default/reinstall the voicemail to factory defaults. Still wish I could find a way to extract the personal voicemail greetings of our users though. Bugs me that Avaya doesn't have a way to export those.

If it's happy I'll rebuild/clone the mirror to a fresh set of drives and toss both of the old disks. They were identical models of Seagate ST3160318AS that we got with our BCM400 to BCM450 raid upgrade kit.
I think I will throw some SSD drives at the thing and see if they work.

Gotta find one of those adapters that fits 2.5 drives with the drive sled in the BCM450 though.
I have a pair of 180GB drives but the included desktop bracket will not position the drive correctly.

 

[QueBall780]

Bah! Have to reset the settings.

Bunch of mailboxes are still having glitches.
So yeah, dead hard drive was likely the cause but it's created some damage.

 

[QueBall780]

Update:

Installed new SSD drive. I have 2 but the second one did not work due to a crappy 2.5 to 3.5 bay adapter product manufactured by SilverStone model number is SDP09
The soldering looks like an 8 year old did it. Very crappy going to need to order a better product this is the only one I could find in stock on local shelves that would allow the sata connector to line up in the drive sled of the BCM450. Most of the available adapters would assume the drive is connected with cables.

But other than that, I used an Intel 335 180GB drive.
The BCM450 seems to support a maximum of 127GB so it does not see the full amount of space available which is fine. More room for the drive to do wear level balancing.

I imaged the drive with the 6.0 image from the Avaya download site using my acronis trueimage 2013 software and it had no problem booting up.
Use the serial port to configure network settings 115200-8n1
Connect with element manager and set the timezone
Patch the system up to the same system update and desktop update as before
Restore the backup

Then to reset the voicemail I do Feature 982 with the operator password and use the reinstall command to reset the call pilot back to factory defaults. Hopefully that's going to fix the problems.
I have recreated or imported all the company greetings, CCR tables, holidays, and mailboxes from my previous notes.

For most users I'm going to make them reinitialize their mailboxes instead of restoring the saved recorded names I made. Lots of the recordings are old and poor.
I initialized a few mailboxes for users in remote branches so the receptionist will not have a problem transferring callers to those voicemail boxes in the morning.