Remote carrier superloop getting disabled

[issdar]

I have a remote carrier superloop that is getting disabled every couple weeks or so. I had AT&T perform test on the T1 lines found no errors, having them perform intrusive test today. Also recieving the CRI3122 EXTERNAL ALARM CLEAR/SET error. Any further troubleshooting I can test on the XPEC?

 

[NortelGuy1979]

You can log into both the far-end controller as well as the local network card and pull stats from the link. Kind of like PM stats from an Adtran T1 smartjack. I bet AT&T finds errors though; make sure they test towards both the A end and Z end, run pattern testing against both CSU's.

Matthew - Technical Support Engineer

 

[SpelingChampeon]

Another suggestion is to see if there is any correlation between the times your CR goes down, and the time they are testing the circuit. One of the problems with telco testing, is that they run a quick circuit test, but in small durations. Most of the time it has nothing to do with YOUR scenario.

Last but not least, ensure you have proper grounding.

 

[issdar]

Thanks for the input. How do you pull stats from both the controllers? I will find out today if AT&T found any errors both directions. I believe in the past they ran a pattern test and found nothing wrong. Could it possible be a bad local or remote controller card? Or card models are not fully compatibly with release 6.0? SpelingChampeon what did you mean by see if the CR goes down? During the intrusive test they are going to take down the circuit during the test.

 

[NortelGuy1979]

CR = carrier remote (which is what you have).

The commands for the MMI are below; the individual letters are the command, IE, "A D" is a command; C A L is a command, C A L D is a command, etc. There's a help on the MMI too I think.

A D Alarm Disable — Disables all local and remote carrier alarms. Maintenance
LED lit.
A E Alarm Enable — Enables all local and remote carrier alarms.
C A Clear Alarm — Clears all local and remote carrier alarms.
C A L Clear Alarm Log — Clears near-end alarm log.
C A L D Clear Alarm Log Distant — Clears distant-end alarm log.
C E Clear Error — Clears near-end error counters for all carriers.
C E D Clear Error Distant — Clears distant-end error counters for all carriers.
C M Clear Memory — Restores all configuration settings on the near-end card to
their original default values.
C M D Clear Memory Distant — Restores all configuration settings on the
distant-end card to their original default values.
D A Display Alarm Log — Displays near-end alarm log.
D A D Display Alarm Log Distant — Displays distant-end alarm log.
D C Display Configuration — Displays current configuration of the near-end
mini-carrier interface cards.
D C D Display Configuration Distant — Displays current configuration of the
distant-end mini-carrier interface cards.
D H Display History — Displays near-end performance counters for a specified
carrier for the last 24 hours.
D H D Display History Distant — Displays distant-end performance counters for a
specified carrier for the last 24 hours.
D M Display Map — Displays the in-service or out-of-service (I or O) status of all
channels on both T1 spans at the near end.
D M D Display Map Distant — Displays the in-service or out-of-service (I or O)
status of all channels on both T1 spans at the distant end.
D P Display Performance — Displays near-end performance counters for the
specified carrier during the past hour.
D P D Display Performance Distant — Displays distant-end performance counters
for the specified carrier during the past hour.
D S Display Status — Displays the status of the carriers on the near-end.
D S D Display Status Distant — Displays the status of the carriers on the distant
end.
E Emergency stand alone mode — Displays current ESAM status and
available ESAM setting options.
E F Emergency stand alone mode, Force — Causes the specified card to
operate in the Emergency Stand-Alone Mode at all times.
E N Emergency stand-alone mode, Never — Disables the specified card’s ability
to operate in the Emergency Stand-Alone Mode.
E A Emergency stand-alone mode, Auto — Causes the specified card to operate
in the Emergency Stand-Alone Mode automatically when two channels go into
alarm.
HE or ? Help — Displays the help screen.
HO Host — Connects to the SDI port. The password is HOST. (@@@ terminates
the connection. See "@@@" entry, below.)
L Log in — Logs into the MMI terminal when the system has one Mini-Carrier
Interface card. The password is MCRLINK. If logged in, typing L and an
incorrect password will log out the user from MCR MMI system.
L xx Log in ( daisy chain address) — Logs into the specified card when the
system has multiple Mini-Carrier Interface cards daisy-chained in the network
module. The password is MCRLINK.
P D Provision Download — Downloads provisioning data stored in the near-end’s
Flash module.
P D Provision Download Distant — Downloads provisioning data stored in the
near-end card’s Flash module to the far-end card’s Flash module.
P E Provision Erase — Erases and tests the provisioning section of the Flash
memory.
P I Provision Import — Requests the near-end card to load its provisioning
data from the Flash memory into RAM, on-board non-volatile memory and
battery-backed RAM.
P R Provision Refresh — Ensures that the LMI and RMI RAM contain all the
provisioning data from every card in the system.
P S Provision Save — Programs the Flash memory with a copy of the system’s
provisioning. This includes serial numbers, site names, ESAM, T1 span
timeslots, enabled or disabled links, and alarm setting
P U Provision Upload — Updates a provisioning file to the near-end card’s FMM
P U D Provision Upload Distant — Updates a provisioning file to the far-end card’s
FMM.
Q Quit — Logs out the terminal user.
S A Set Alarms — Allows user to set carrier alarm threshold and duration
parameters.
S C Set Configuration — Allows user to set carrier configuration, such as
equipped or unequipped.
S D Set Date — Allows user to set or verify the date currently maintained by
near-end mini-carrier interface cards.
S D Set Date Distant — Allows user to set or verify the date currently maintained
by distant-end mini-carrier interface cards.
S E Set ESAM — Allows user to configure the parameters of the Emergency
Stand-Alone Mode.
S N Set Name — Allows user to set the site name.
S S Set Span — Allows user to set the number of timeslots used by the T1.
S T Set Time — Allows user to set or verify the time currently maintained by
near-end mini-carrier interface cards.
S T D Set Time Distant — Allows user to set or verify the time currently maintained
by distant-end mini-carrier interface cards.
@@@ Terminates the terminal connection to the host.
T Test — Allows user to select 1 of 5 tests to be run on the specified carrier link.
T D Test Distant — Allows user to select 1 of 5 tests to be run on the specified
carrier link at the distant end.
T V Test Verbose — Instructs system to display error reports while tests of
specified carrier link at the near end are in progress.

Matthew - Technical Support Engineer

 

[issdar]

Thanks for the reply Matt those commands look like they would be helpful although I am unfamiar with MMI. Did some searching and can't find exactly how to access the MMI terminal.

 

[NortelGuy1979]

Do you have NN43021-555? There's a 9-pin serial connector on your I/O panel that is for the carrier remote superlooop / network card. The baud rate is usually either 1200 or 2400, 7-M-1 I believe. It's set via DIP Switches on the card.

MMI mode
In the MMI mode, a terminal or TTY is connected to the local MMI port at
the Local Carrier Interface card and another terminal or TTY is connected
to the MMI port at the Carrier Remote IPE as described in Installing SDI
and TTY cables in this chapter. Each terminal controls the local MMI
functions of the card it is connected to and some display functions at the
distant site.
The default MMI interface characteristics are set in the Local and Remote
Carrier Interface card EEPROM as follows:
• Speed: 2400 bps (speed is an exception, it is selected using DIP SW1
position 1 where OFF = 2400 bps and ON = 1200 bps)
• Character width: 7
• Parity bit: mark
• Stop bit: 1
The MMI mode is accessed from the initial (INIT) mode by entering L on
the MMI terminal or TTY to log in. For multiple Local Carrier Interface
cards connected in a daisy-chain, to log in to the specified Local Carrier
Interface card, enter L <sl>, where sl is the address and the superloop
number of the specified superloop, as shown in and .
Password is M1LINK
The MMI command line prompt in the initial mode is: CRI>, but after
logging in, the prompt changes to: CRI::>

(from the NTP); I think you can download that one from Nortel.

Matthew - Technical Support Engineer

 

[issdar]

Thanks! I will give it a try and also let you know what the card model is when I go on site.

 

[issdar]

The card model is NT7R52AD and I did see the 9-pin connector wasn't able to try to logging though. I am very suspicious that both T1 circuits could be bad.

 

[NortelGuy1979]

I googled T1 test patterns; found this PDF on a website. No idea who wrote it, but it gives you the standard patterns they run. Make sure they run more than 1 pattern and run for extended periods of time; see if you can get PM stats on the circuit, see if they'll run pattern not only against the smartjack but also the CSU on both ends. (Some telcos will, some won't...)

Matthew - Technical Support Engineer

 

[issdar]

Thank you Matt I was able to log into the MMI terminal and on the 1st T1 local end there is 3000+ error seconds 3000+ bursty seconds and 65000 error counter! The performace log every few seconds there is performance within threshold entry and then level 1 alarm active bit error rate. But the 2nd T1 there was hardly any errors. I cleared the logs and performance counters. I will see what AT&T test results come up with tonight. I believe they are running 30 min test patterns per side. What's your take on it?

 

[issdar]

AT&T reported no errors on the first T1 although second pne yet to be tested. By viewing the log in MMI it Shows that the second T1 has been down since last night , good thing for redundancy. Check the adtran T1 card and there is red lights w/ card being bounced. AT&T said tech currently working on it must be in a manhole or something.

 

[issdar]

So AT&T came out and somehow "magically" they managed to get the T1 card to come up with green lights and did several test patterns to T1 card that checked out. But unfortunately the T1 circuit is infact not operational and on both remote and local carrier cards there is alarm level 3 out of frame tried bouncing loop and rebooting the whole remote pbx no luck. Going to have AT&T run several pattern test to the local carrier side of PBX to make sure that card is good. So frustrated right now going to have to take break on it and use sunset T1 test meter to Loopback one side of the T1 and run pattern test myself end to end. Sighhhh