Symposium NCC cause problems?

[picklemogg]

Hey all- any thoughts would be awesome. We have 5 SCCS 5.0 networked sites via NCC 5.0. We had an issue where one of our sites lost power to the PBX and SCCS 5.0 server. At the time of the power outage our 4 other sites within the Symposium network started having problems communicating locally across the ELAN to the PBX- receiving CSA106 errors (agts couldnt login, etc.). All 4 sites have point to point T1s connected to site that lost power. During the time of power loss we were extremely busy and I'm sure every site had current network calls up across T1s to the site that lost power. In addition the site that lost power would have had current calls queued up to network agents at all 4 other sites. So basically having a power outage in 1 location caused all other 4 SCCSs to fail. We had to restart all to bring back in service 100%. So 1 of 2 things may have caused issue which neither make sense to me. Calls that were currently up across tielines were dropped after power outage and T1/d-channel alarms caused probs or the fact that any site that had calls queued to from the site that lost power had probs. Like the network SCCSs couldnt send a response back to the SCCS that sent req for net agent?

Any ideas?

Thanks,
PMoggs

 

[mikeyb123]

Alot of this will depend on where your calls originated, how you are routing calls to the other sites, etc Do the other sites have direct links to each other or do they converge on the one site that lost power?

How does you scripting work?

Loads of questions to be looked at....

It's not getting any smarter out there. You have to come to terms with stupidity, and make it work for you.

 

[picklemogg]

Mikeyb123- agreed many ways to set this up. That being said, if any possible setup can cause 5 SCCSs to lock up because 1 lost power what you have is an absolute horrible technical design and flaw.

After Nortel dug around it was determined that because current network calls were queued out from site that lost power prior to outage network nodes could not respond back to power stricken site with reserved agent responses. Because of this, a process called agent supervisory module locked up. After this process was hosed, a domino effect followed and soon after SCCS was having problems communicating across ELAN link to local PBX at all sites. Nortel has written a patch for us so hopefully this problem doesn't happen again. We'll soon be on CCMS 6.0 so it should be a thing of the past according to Nortel anyway since it is fixed in 6.0.

 

[Milesprower]

Wow, Nortel should have firewalled this issue a long time ago. In a network environment, you need to run through all the scenarios that can happen and mitigate their impact accross the network.

I suspect it might originally have been designed correctly, but was broke when new features were developed. Networking has been around since Version 3 - this is the first time I have heard of a single site power outage taking down the whole network.

 

[picklemogg]

Yeah- Nortel was able to reproduce in their lab. They later told us that 2 other customers had a similar problem-neither were in the US.