BCM50 R3 that keeps crashing every 46-60 days and has been replaced 3-times already!

[RodneyMcSnow]

I have a client with a BCM50 R3 System Software Version 6.0.2.05.237 with 10-T7316e phones and 8-Analog Trunks with Basic Contact Center and Reporting.
Every 45-60 days the Nortel BCM50 System locks up and crashes and we have replaced the system 3-times since March of 2014 and Avaya tells me that the customers business is putting to much stress on the system and overloading the CPU on the BCM50. I DONT BUY IT!

The customer gets between 300-600 calls a day Monday through Saturday and there are 4-Agents in the Call Center. The iTEL Reporting for Contact Center software is run on a Dell i5, 64-bit Computer and we have had no problems with the computer, just the BCM50.

We have had an electrician out on site to verify that the system is on its own dedicated electrical circuit with isolated ground that connects to an APC SMART 3000 XLM UPS that only the BCM50 with 2-expansion cabinets and the Dell Reporting computer are plugged into.
The programming has been hand input, each time just to rule out database corruption. All patches are current and up to date, the APC UPS has been tested with APC to verify that its ok.
Other than replacing with an Avaya IP Office, I am out of options and open to suggestions, anyone have any ideas or recommendations?

 

[curlycord]

Throwing some stuff out here.....

-You should run Monitor to see CPU usage etc.
-Also did you check the alarms? any codes to report?
-did you replace the analog module?
-did you replace the expansion cabinet?
-test all lines for voltages (idle and ring)
-program fresh or always restoring a backup?





=----(((((((((()----=
curlycord

http://www.curlycord.com

Toronto Canada

http://www.telcopc.ca

 

[RodneyMcSnow]

-You should run Monitor to see CPU usage etc.
I let the monitor run and it appears to be running as low as 6% but most of the time it is 50% or higher

-Also did you check the alarms? any codes to report?
There are no current alarms or logs other than normal system info at this time, system has been online for about 18-days.

-did you replace the analog module?
No, did not replace, why would it need to be replaced?

-did you replace the expansion cabinet?
No, did not replace, why would it need to be replaced?

-test all lines for voltages (idle and ring)
Will perform this today. Have to pickup some batteries for my test equipment.

-program fresh or always restoring a backup?
Set to backup monthly. Sine there has never been a change from the very first install, I am going to turn off for now and perform manual backups as needed.

 

[Cyberprog]

I've seen issues before caused by expansion cabinets, had one in Swindon where I ended up swapping the cabinet, psu and brim and problem was solved. Dunno which of the bits was the fault of course, but they're so cheap...

Alex Threlfall
Cyberprog New Media

http://www.cyberprog.net

Telecoms, Networks, Hosting, Alarms, CCTV etc.

 

[curlycord]

I have seen CO's or Line cards or module etc cause havoc on systems before.

As for restore I meant when you changed the system did you do a restore from a backup or re-program from scratch....if there is a glitch in the software then you might be backing it up too but just a thought.

BCM50 taxed with only 4 people and 8 lines seems nuts, I would have asked them how they came to that conclusion.....even so if anything I mention was in fact the issue they may not have been able to detect it anyway.

Not sure but there is probably a way to check logs for any CPU issues at times of crashes.


=----(((((((((()----=
curlycord

http://www.curlycord.com

Toronto Canada

http://www.telcopc.ca

 

[snowman50]

If you have changed the ccu 3times already then I would go for changing out the other add on bits.
As stated there must be some other reason why it reboots.

Do you have out of memory alarms as thes can caused problems.

Also has Rcc been installed twice or on another machine.
This will also cause the system to freeze and reboot.

 

[ucxguy]

The CPU usage of 50% (or a bit more) with just a few users seems to be a bit suspicious, but is not likely to cause any problems. If it was 95% or more over extended periods of time, that would be something else.

There probably is a memory leak in some BCM software component that causes the system to run out of memory and "crash" every 45-60 days. If you use BCM Monitor to monitor the system, do you see the memory usage going up over the 45 days? Also, could you describe symptoms of the "crash"?

A simple workaround might be to reboot the BCM50 system every 30 days. This workaround is simpler than changing various pieces of hardware. If the problem is the BCM software (and that is pretty likely), hardware changes won't help.

You could maybe try another BCM50 with the software release 5 or 6. If the cause of your problem is really a memory leak, there is a chance the problem was fixed in one of the more recent software releases...

 

[Firebird Scrambler]

You mentioned that the latest patches had been applied. Were they as shown below?.

BCM50 3.0
==================================

PATCH ID: BCM050.R300.SU.Desktop-265.201110 Category: GEN
PATCH ID: BCM50R3_FactoryImageUpgrade Category: GEN
PATCH ID: BCM050.R300.UBOOT-214 Category: GEN
PATCH ID: BCM050.R300.AMP-COLUMBIA-177 Category: LTD
PATCH ID: BCM050.R300.AMP-SOUTH-KOREA-149 Category: LTD
PATCH ID: BCM050.R300.SOFTWARE-MANAGEMENT-63 Category: GEN
PATCH ID: BCM050.R300.SU.System-270.201205 Category: GEN

Another thing is if the backups increase alot each week. The BCM backups will usually increase as they contain the CDR data. I don't actually know what the maximum backup file can be, but I've seen some pretty large ones over the years. They should compress, but as your system is crashing around the 40 day period, then I would question the backup size?. When you rebuild the system, you would not have any CDR data to input.

I'm assuming that your hard disk is the 80 gig Seagate one?. Set up a weekly backup to another device such as a USB or other computer and compare the backup sizes to see how much the files have increased.



All the best

Firebird Scrambler
Meridian 1 / Succession and BCM / Norstar Programmer in the UK

http://www.linkedin.com/in/davidbromley

If it's working, then leave it alone!.

 

[Cyberprog]

It's always handy to run a CDR Pull with the delete switch. If you fill the partition, it screws the BCM up and you have to do a reset (or get avaya to SSH in and delete the files).

Alex Threlfall
Cyberprog New Media

http://www.cyberprog.net

Telecoms, Networks, Hosting, Alarms, CCTV etc.