Your worst outage(s) ever and what caused it

[phadobas]

I'd like to know what was your most memorable (?) outage on a NEAX 2400 or 2000. I'd like to know also know what was its cause. Reason being is I have both of these systems under my care and the 2400ICS is getting old, and would like to know what to watch out for mainly to avoid any sort of outage.

Appreciate all information.

 

[CDRMAN]

Normally you will find the products will work fine on their own. They can run for many years.
The issues will come up when you start playing around and cause your own outage.

Always backup the systems after programming at least a weekly backup an format.
Make sure your UPS is up and running
Don’t mess with programming you do not fully understand what you are doing.

Upgrade your ICS to IMX – NEC may still have programs to do at a cheaper rate.

 

[phadobas]

Thanks for the good advices. I found all of your information true:
The ICS has been running since 1999 without a glitch
I do backups every week
UPS is being checked once a month
Upgrade to IPX is scheduled within a year

 

[belevedere]

On an old 2400 ICS if you have the upright CPU's, consider upgrading soon. You cannot get them anymore. If you are not fully redundant (CPU,Time Division and Power Card) then consider adding it. The worst outage we have seen is water damage (burst pipes) and a direct lightning hit.
On the 2000, Power surges and lightning hits.

 

[mextera]

You will find that NEC's MTBF is some of the best in the industry and real numbers are better that what is published!

Outages on the 2000 have mostly been power related lack of it or too much providing surge. Protection is a key.

Mextera

 

[phadobas]

For the 2000IPS, I have to have strong protection since I'm in the lightning capital of the planet: mid Florida, Tampa bay...

Anyways, I have all my T1-s coming in through fiber to the building, so I should be fine...

 

[smoom]

Worst one for me....Customer wanted a show piece. Switch room was 40' x 40' wrapped floor to ceiling with static guard carpet. Large glass windows so anyone walking by could see the room. We installed a new outside plant for voice, video, and data, and put a new 2400 IMX in. 12000 phones and 8 PRi's 4 T1's everything was redundant....But 3 days after cutover lightning came through the window hit the system in the backplane, it fried everything including the redundant equipment. Even the hard drive on the voicemail melted. It took us 4 days to get new equipment and get it installed.

 

[phadobas]

Now that's a solid one. I guess it's hard to protect against that one. Except for putting the system in the basement...

 

[mextera]

Then you have the flood's water is awful to look at when halfway up a cabinet.

Mextera.

p.s. look at

http://www.2000stuff.com

He ( I say He very sexist of me) has a pictures link very good a must look.

 

[gbaughma]

I think that fixin will agree with me.

Hotel/Casino... about 350 phones. 6 PIMs. 2000 system.

Lightning strike came in from the single-line station at the golf course clubhouse.

Pretty much took out the entire system, including 5 of the 6 power supplies, a couple of the bus cards, 4 or 5 DTerm station cards, several single-line station cards, and (if I remember right) the CPU.

It worked out to about a week of being up there....

I still remember the call... "(resort/casino name) called. They're having a problem with their phones... can you run up and check it out?"

... and the award for Understatement of the Year goes to.....



Just my 2¢

"In order to start solving a problem, one must first identify its owner." --Me
--Greg

http://parallel.tzo.com

 

[phadobas]

Lightning is pretty sneaky. It finds all possible ways to get into your system...

I had all trunks protected, and the system (some channel bank) got still hit through a single line station going outdoors. Luckily, only one card got affected, but it teaches you a lesson.

 

[OzzieGeorge]

My worst outage only lasted a few seconds....
Someone could have told me the suicide psychiatric help line was through the PABX before I reset it!!!!

"I'm Going to Jump I tell you....Hello....Hello"

 

[gbaughma]

OzzieGeorge:

Can't be any worse than calling the Suicide Hotline, getting the automated attendant to wait for the next counselor, and the music on hold is "Don't Fear the Reaper" by Blue Oyster Cult.



Just my 2¢

"In order to start solving a problem, one must first identify its owner." --Me
--Greg

http://parallel.tzo.com

 

[ethereal112]

ChinoLee,

1.5 stack with 1100 users including VoIp,Middle of February no lightning storms in the desert (Tucson, AZ) com ports on our Octel voicemail server, Dell server for Call accounting, and NEC 2400IPX LAN connection for Mat Terminal all burnt out. ESD (Electro Static Discharge) was the criminal. Costly one to because Avaya had bought out Octel and discontinued the voicemail server, We did find a reseller but it was 14K just for one circuit card. Everything else was on a maintenance agreement.

 

[mextera]

Wow 14K why not just get the AD120 with Unified messaging.

Mextera

 

[ethereal112]

The time it would of taken to get the order in on top of that installation and implementations, downtime would have been to great as oppose to resetting a new card and service restored less than 2 days. Very impatient user's!