Digital sets go dead on CS1000E if any PRI goes down.

[Wedger]

Small CS1000E Rls 5 with 2 cabinets. (Cabs not chasis)
If any PRI goes down (1st loop and then DCH)then all digital sets on cab housing PRI go dead for 20 mins approx and stay dead even if PRI is restored.
If affected PRI and DCH is disabled in software then fault clears.
Only occurs if PRI goes down externally.
Will not occur if PRI is brought down in software.
Analog sets in this cab ok during fault.
All othrr cabs ok during fault

Have applied latest deplists.
Have replaced all PRIs/DCH/CC cards
Have replaced MGC card in affected cab.
Have replaced PSU in affected cab.
Stripped cab down to the following:
0 MGC/1 PRI/2 -/3 CPPM CS/4 Dig - left all other card/cabs out of service.
Tried on another system in lab - could not reproduce.

Affected cab normal layout
MGC/PRI1/PRI2/Dig/Dig/Dig/Dig/Falc/Falc/CP-PM CS/CP=PM SS.
This system was upgraded from Meridian Rls 25.15.

Anybody come across this issue?

 

[johnpoole]

have not heard of that one here.. sounds like a clocking problem.. are you clocking off of that pri.. when the pri goes down do you take lcnt hits prior to oos?

john poole
bellsouth business
columbia,sc

 

[Wedger]

Hi John

Sorry for the delayed response - I'm juggling a few balls like this one at the moment.

System is clocking from the Pri in slot 1, however digitals are affected if either Pri in slots 1 or 2 go down.
Should have pointed out that I get no errors whatsoever on
HST, PDT or the MGC Cli other than the normal DTA errors when I break either Pri loop followed by the normal DCH errors/messages when the D Channel de-establishes.
As soon as the D Channel drops the digital sets go crazy.

Have now proved all hardware involved including the cabinet except for the CP-PM CS. Don't have one available at present but this card is also driving 2nd cab with no issue?
Tried main equipment on a separate cabinet with 1 digital card and all hardware/cables not required removed from system - still have issue.
Also tried shifting all IP to a dedicated switch that was off the network - no joy.

I'm beginning to think this might be a software database issue as the database was migrated from the old rls 25.15 system.
Needed to step through Rls 25.40 when migrating as we were getting corruption on the Dect sets in the 2nd cabinet if we migrated straight from 25.15. to Rls 5.

No hair now left!!

 

[allenmac]

Are your digital sets at the latest firmware? Have you tried forcing a download to a couple of sets to see if that makes any difference?

Just my $.02

 

[Wedger]

Theres a mixture of M3900 sets and also a a number of M3310/M3820 sets.
Have tried new M3900 sets straight out of boxes.
Know ther was an issue with M3902 firmware but not aware of issues with M3903/M3904.
Same issue with all sets.
This site is networked to other sites using SIP trks and some of thes sites have been brought up from various levels of release and do not have this issue.
Mine is the only site with this issue.

 

[curtismo]

Can you describe "dead" a bit further?

- phone displays dark or lit?
- no dial tone on phones during problem?
- Can you call other phones in same cab?
- Type of cards for PRI (TMDI)?
- Tried to change clock daughterboard?

Have you swapped the clock daughterboard and/or the TMDI/other card the clock card is on?

 

[Wedger]

Mostly loose control over Digital sets.
Sometimes loose setup and tear down contol of calls.
Most times you can get dial tone but can't do anything with it.(Checked DTRs and found ok.)
Line icon beside SCR key lights and stays lit even if you hang up.
Sometimes displays go completely dead.
An existing speech connection will stay up but you are unable to clear the call when complete.
Just seem to loose signaling control of calls.

Using NTDK50 PRI cards.
Tried replacing all PRIs/Clock Controller/DCH cards as mentioned above - no joy.
You can try ringing one of the digital sets from an analog set in the same cab but you just receive ringback on the 500 and no ringing on the digital set.
Analog sets in the affected cab stay working - as mention above.

 

[curtismo]

You know, it sounds like the only two items you haven't replaced are the cabinet or the power supply. There was an issue at some point with MGC, T-1s and Opt 11 cabinets that caused noise during calls and NT had a stop-ship order out until that got resolved; maybe some sort of noise on the backplane is causing issues. When you are telling us the display is going dead that's telling me the switch is no longer processing (which in this case, the MGC card is the processor for the cabinet).


BTW, the NTDK50 is the cabinet; the TMDI card is NTRB21; NTAK09 is the older DTI/PRI card for Option 11.

 

[curtismo]

...to add, I'd think at this time you have to get Nortel involved. Something ain't right here. You know if you pull T-1 cards out without disabling the clock card, the switch pretty much will INI; and in this case this cabinet (MGC) is having problems with clocking when the PRI goes down.

 

[curtismo]

...add 2

I heard you say "LAB", do you have a spare cabinet and power supply. I'd see whether you could try moving the two CP-PM units out into another cabinet. Not sure, but I don't think you need to have a MGC in this cab; I believe the CP-PM only needs the backplane for power.

 

[tekice]

upgrade to the latest loadware on the mgc card loadware 46 and uprade bios on cppm to rls 15 should clear your issues

 

[Wedger]

Thanks to all for your various inputs.

I think I have nailed the issue.
I took a copy of the database offsite and loaded it on another system.
Was able to replicate the issue on test system.

Stripped out all datafill except common CDB/CFN info etc.
Left 1 PRI and 1 M3904 in d/b and problem still existed.
Removed all PRI datafill including DCH/DIDs/Clocking info down to CEQU setting on PRI and reprogrammed again - issue cleared ???
Problem with this is that this datafill was not migrated during the upgrade but needed to be added manually.
Might have suspected some form of error input during programming but compared snapshots of info removed to info replaced with "Beyond Compare" and could find no differences but maybe after 38yrs at this game I'm finally losing it.
Suspect the removal and reprogramming of clocking info in LD73 might prove to be the culprit.
Hope to nail it down further when I return to affected site. Will post any further findings.

 

[curtismo]

If this isn't it, tekice I think is correct. If you have access to the Nortel Knowledgebase there is a solution out there from a month ago, with one of the symptoms being no dialtone on digital sets.