BCM400 4.0 - Still Causing Issues

[TekTRADERS]

Good Morning,

Went into my clients last evening and replaced the power supply and hard drive on a BCM400 4.0 and applied the last available patches. Upgrade went great! My client was having issues with the voicemail and IP sets going down (Hard Drive I thought) maybe once a week.

Client calls me this morning and tells me that the IP sets and Featue 981 is not working again.... Everything was good at midnight before leaving the job site. The strange this is that the customer is still able to enter the system via Element Manger and do a soft reboot.

Do I have a bad MSC (hope not) of something with my PECIII cards?? I know we can't transfer to licenses onto a new MSC card now! Yikes

Help....any assistance or thoughts would be appreciated.


"The bitterness of poor quality is remembered long after the sweetness of a low price has faded from memory".

http://www.tek-traders.com

 

[exsmogger]

The MSC operates the telephony side of the BCM 400. The hard drive and base tray operate the voicemail and data side. My guess would be that you have a failing base tray with the unfortunate problem of leaky capacitors.

Brian Cox
Georgia Telephone

http://Georgia-Telephone.com

http://www.linkedin.com/in/briancox1952

 

[telcodog]

You weren't very specific about access. Can you access the voicemail via element manager or directly? Did you ensure that the vm is initialised. If you changed the hard drive, you will need to do a restore from a backup to get your voicemail back!

 

[TekTRADERS]

Good Day,

The system is rebooting after 45 min or so. At first the client indicated that is was just the voicemail/call center/and IP sets were going, but as of Thursday he indicated that the whole system was going down and then coming back. So, the system is rebooting itself. I returned to site Thursday at 5:30pm and replaced the base function tray. The client was hanging around to finish some paper work and again I received a telephone call indicating the system rebooted itself again.....

To date I have replaced:

1 - Power Supply
1 - Hard drive (restored data OK)
1 - BFT

I think there may be an issue with one of the MBM. System consists of PRI, 2 - DMS32 and 1 - 4X16

These are old original cards from 2004 and maybe one of the cards is drawing too much current or something.... Any thoughts?

I think the MBM's are the only thing left in the equation. It's its the MSC then the client is pooched....

Cheers


"The bitterness of poor quality is remembered long after the sweetness of a low price has faded from memory".

http://www.tek-traders.com

 

[exsmogger]

Have you checked the alarm log? Maybe it will show a major or critical alarm just before each reboot.

Brian Cox
Georgia Telephone

http://Georgia-Telephone.com

http://www.linkedin.com/in/briancox1952

 

[TekTRADERS]

Good Day Brian,

Here are the alarms... reading from the bottom up. This line appears to start the shutting down of services "**ERROR** Received EMS_EVENT_CONNECTION_LOST from MSM;MPS service aborted" The MSM; MPS


Fri May 09 12:13:14 EDT 2014 false 10228 warning Service Manager - Integrated Wan (Wan) has been stopped either due to user action or because Service Manager has stopped this service due to a dependency on another service that has been stopped. This will affect the ability to use ISDN dial up services on the system. systemId=BCM;entityId=BCM40;entitySubId=monit

Fri May 09 12:13:09 EDT 2014 false 10202 warning Service Manager - CallPilot has been stopped either due to user action or because Service Manager has stopped this service due to a dependency on another service that has been stopped. systemId=BCM;entityId=BCM40;entitySubId=monit

Fri May 09 12:13:07 EDT 2014 false 10230 warning Service Manager - Reporting for Contact Center Service (CCRSAppServer) has been stopped either due to user action or because Service Manager has stopped this service due to a dependency on another service that has been stopped. This will affect the ability to use Reporting for Contact Center systemId=BCM;entityId=BCM40;entitySubId=monit

Fri May 09 12:12:44 EDT 2014 false 10210 warning Service Manager - System Set Based Admin Feature9*8 (ssba) has been stopped either due to user action or because Service Manager has stopped this service due to a dependency on another service that has been stopped. systemId=BCM;entityId=BCM40;entitySubId=monit

Fri May 09 12:12:42 EDT 2014 false 10211 warning Service Manager - Computer Telephony Service (Cte) has been stopped either due to user action or because Service Manager has stopped this service due to a dependency on another service that has been stopped. This will affect LAN CTE and the Line Monitor in BCM Monitor. systemId=BCM;entityId=BCM40;entitySubId=monit

Fri May 09 12:12:40 EDT 2014 false 10229 warning Service Manager - Doorphone service (BCM_Doorphone) has been stopped either due to user action or because Service Manager has stopped this service due to a dependency on another service that has been stopped. This will affect the ability to use a doorphone on the system. systemId=BCM;entityId=BCM40;entitySubId=monit

Fri May 09 12:12:36 EDT 2014 false 10212 warning Service Manager - Line Monitor Service (lms) has been stopped either due to user action or because Service Manager has stopped this service due to a dependency on another service that has been stopped. This will affect the Line Monitor in BCM Monitor. systemId=BCM;entityId=BCM40;entitySubId=monit

Fri May 09 12:12:34 EDT 2014 false 10206 warning Service Manager - Quality of Service Monitor (qmond) has been stopped either due to user action or because Service Manager has stopped this service due to a dependency on another service that has been stopped. systemId=BCM;entityId=BCM40;entitySubId=monit

Fri May 09 12:12:33 EDT 2014 false 10226 warning Service Manager - CoreTel Logging (LogManagement) has been stopped either due to user action or because Service Manager has stopped this service due to a dependency on another service that has been stopped. This will affect the logging of CoreTel. systemId=BCM;entityId=BCM40;entitySubId=monit

Fri May 09 12:12:32 EDT 2014 false 10225 warning Service Manager - IP Terminal Firmware upload Service (UftpServer) has been stopped either due to user action or because Service Manager has stopped this service due to a dependency on another service that has been stopped. This will affect the ability to download new firmware to IP terminals. systemId=BCM;entityId=BCM40;entitySubId=monit

Fri May 09 12:12:31 EDT 2014 false 10224 warning Service Manager - IP Terminal Service (EchoServer) has been stopped either due to user action or because Service Manager has stopped this service due to a dependency on another service that has been stopped. This will affect IP terminals from operating properly. systemId=BCM;entityId=BCM40;entitySubId=monit

Fri May 09 12:12:30 EDT 2014 false 10233 warning Service Manager - IP Music Service (ToneSrvr) has been stopped either due to user action or because Service Manager has stopped this service due to a dependency on another service that has been stopped. This will affect the ability to use IP music. systemId=BCM;entityId=BCM40;entitySubId=monit

Fri May 09 12:12:29 EDT 2014 false 10232 warning Service Manager - IP Music Service (BcmAmp) has been stopped either due to user action or because Service Manager has stopped this service due to a dependency on another service that has been stopped. This will affect the ability to use IP music. systemId=BCM;entityId=BCM40;entitySubId=monit

Fri May 09 12:12:25 EDT 2014 false 56006 minor IpTelProvider shutting down due to fatal error. systemId=BCM;entityId=BCM40;entitySubId=IpTelProviderAgent

Fri May 09 12:12:24 EDT 2014 false 53004 critical Mgs:: Shutting down due to MSM communication failure systemId=BCM;entityId=BCM40;entitySubId=mgs

Fri May 09 12:12:24 EDT 2014 false 53005 critical Mgs:: Shutting down due to MPS communication failure systemId=BCM;entityId=BCM40;entitySubId=mgs

Fri May 09 12:12:24 EDT 2014 false 50064 critical OAMDataSend 29 29:
OAMDataSend 29 29:
OAMDataSend 29 29:
OAMDataSend 29 29:
OAMDataSend 29 29:
OAMDataSend 29 29:
OAMDataSend 29 29:
The Media Path Management sub-system unexpectedly became offline - terminating systemId=BCM;entityId=BCM40;entitySubId=utps

Fri May 09 12:12:24 EDT 2014 false 52006 critical **ERROR** Received EMS_EVENT_CONNECTION_LOST from MSM;MPS service aborted systemId=BCM;entityId=BCM40;entitySubId=mps

Fri May 09 12:12:08 EDT 2014 false 52005 critical **ERROR** Reset by Network Manager systemId=BCM;entityId=BCM40;entitySubId=mps


"The bitterness of poor quality is remembered long after the sweetness of a low price has faded from memory".

http://www.tek-traders.com

 

[exsmogger]

I've been reading other threads regarding this same problem and it seems to point to the MSC card. I know this isn't what you wanted to hear.

Brian Cox
Georgia Telephone

http://Georgia-Telephone.com

http://www.linkedin.com/in/briancox1952

 

[UCXNortel]

If you have a different BCM 400, you can take the MSC card from the failing system and install it in the replacement system without worrying about having to get the keycodes assigned to the new system because the SYS ID is based on the MSC Card and not the cabinet.

Here is an outline to try:
1. Get the failing system up and IMMEDIATELY do a Backup and save the file to your computer as a "xxxxxxxxxxxx.tar" format. Saving with the ".tar" extension will allow you to open the Backup file and retrieve the keycodes to install in the new cabinet.
2. If you replace the hard drive as well, you MUST be on the SAME level of s\w or the Restore will not work properly!
3. Install the MSC Card into the new cabinet, boot the system. BEFORE you do anything else, make sure you can talk to the system with EM.
4. If you can access the system after component replacement, restore the Backup program file you did in Step 1. (FYI- I always extract the keycode file from the .tar backup and install the keycodes first, reboot and then do a restore of the programming.

If this system is "resource intensive" as in many IP phones, lot of voice mail \Contact Center activity, my initial guess would be the PEC III Cards or a memory leak in the cabinet. The alarms you listed are basically in succession of the initial fault which look like either PEC Cards or memory leak. The MSC Card handles the Telephony part and if this was simply a hard drive issue, the Telephony part may still work, but you would NO IP sets, voice mail, etc.

Hope this helps!

http://www.ucidirect.com

Stocking Distributor for E-Metrotel

 

[TekTRADERS]

Good day

Thanks Brian and UCX

The system is virtually new at this point.

I replaced everything except for the MSC including the pec cards and media bay modules. No difference The system reboots all services except for the telephony anywhere between 45 min up to 2 hours

The MSC is corrupt. No saving this 11 year old system

I am picking up a new Avaya BCM450 tomorrow and migrating the codes from the old switch. I'm going to hand-bomb all of the data back into the new system as I don't want to carry forward any corrupt data. I have other systems available, but the client opted for the 450 even though there is a limited shelf life for this product line.

Bring on the coffee and patience. All night project tomorrow!

Thanks again all!



"The bitterness of poor quality is remembered long after the sweetness of a low price has faded from memory".

http://www.tek-traders.com