2400ICS stopped sending smdr 1

[phadobas]

Never seen such a thing: I have two systems, an sv8500 and a 2400ICS. They are CCIS connected to each other. SMDR recording PC is connected to the 8500 via rs232. SMDR records for calls made on the 2400 are sent to the 8500 via the ccis which in turn, sends them to the PC. This is Centralized Billing... This has been working for over a year without a hitch.
Yesterday afternoon however, I noticed that my SMDR pc only shows calls made on the 8500, but nothing coming from the 2400. Just out of the blue it stopped. Checking the last calls reported by the 2400, and what programming changes were done, revealed nothing useful. The last programming change I made was 2 hours before the problem started, and these changes were made on the 8500, using AFRS and AOPR command (to change the routing of calls starting with certain digits). So, I'm totally baffled. Also, went through the manual to check the programming steps for Centralized Billing and SMDR, but again, found nothing wrong. CCIS link works fine for voice calls, and can't really monitor what goes on on the data channel.
Then, I set up the 2400 to report SMDR on its own serial port instead of the CCIS, and when I connected my PC, it dumped about 10,000 records from its buffer. But, while call records always start with KA for outgoing calls, KE for incoming calls and KB for station-to-station calls, I saw a bunch of records starting with KH?!?!? What is that??? Also, those records seem incomplete. Won't show authorization codes used to make the call.
I know that to debug this, best would be for anybody to be on-site, and this information may be way insufficient to advice on handling, but at this point I take any suggestions as I'm loosing smdr records big time and in an enterprise environment, that's really bad.

Thanks

 

[3racefan3]

KH is outgoing extended format. Check your ASYD settings. Probably indexes 241,288 and 296.

 

[phadobas]

I went ahead and checked index 288. Bit 5 was flagged and I turned it off, which turned off the expanded format of the SMDR. But still no go accross CCIS. So currently, I'm switching the serial connection back and forth between the two systems do download their SMDR while the other is buffering it.
Checking index 241 in the manual doesn't show how it would be related to smdr over CCIS, so I left that alone (9E).

Index 296 on both systems are 01 which means 'Output of Calling Number (ANI) is in service'.

Still open for suggestions.

 

[peporter]

CHeck these commands and verify it is all correct and then attempt to resend your calls to the 8500 via centralized billing. The format below is a little messey on the paste, but you can figure it out.



Center Office
General:
CM 08>800>0 For Internal SMDR
CM 04, YY=01>05>0 Built-in SMDR output to LAN
CM 04, YY=01>08>00 For Extended NEAX 2400 IMS Format (KH & KI records)
>15 For Former NEAX 2400 IMS Format (KA & KE records)
CM 41, Y=0>03 Set the pseudo answer timer.

Parity setting for LAN SMDR (to match LAN Call Accounting Device):
CM 08>827>0 No parity
>1 As per CM08>828 (default)
CM 08>828>0 Odd parity
>1 Even parity (default)

For Outgoing SMDR Records:
CM 13, YY=06>Sta. No.>1 For outgoing SMDR by station
CM 35, YY=14>Route No.>1 For outgoing SMDR by trunk route

For Incoming SMDR Records:
CM 13, YY=05>Sta. No.>0 For incoming SMDR by station
CM 35, YY=49>Route No.>0 For incoming SMDR by trunk route
CM 08>426>0 Effective for all incoming calls
>1 Effective only for incoming calls with Account Code
CM 08>463>0 Send ANI/Caller ID to SMDR
>1 Do not send ANI/Caller ID to SMDR

For Tandem SMDR Records:
CM 08>040>0 Allow tandem SMDR records
CM 08>803>0 Incoming and Outgoing records of tandem call is sent to SMDR
>1 Only Outgoing record of tandem call is sent to SMDR

CCIS Programming:
CM 08>368>0 Provide the system with Centralized Billing for Center office.
CM A7, YY-03>CCH No.>1 Distant end is a Local/Tandem office.
CM A7, YY=04>CCH No.>NONE
CM 04, YY=01>06>0 SMDR from Local Office output to LAN

Local/Tandem Office
General:
CM 08>800>0 Output SMDR to CCIS Center office
CM 04, YY=01>05>7 Built-in SMDR output to RS Port on MP
CM 04, YY=01>07>00 For Extended NEAX 2400 IMS Format (KH & KI records)
>15 For Former NEAX 2400 IMS Format (KA & KE records)
CM 41, Y=0>03 Set the pseudo answer timer.
NOTE:
CM 40, YY=00>0 & 1> MUST NOT be set as 11 or 14.
Local office in a CCIS network can not output SMDR from the local RS port
on the MP when Centralized billing is used.
CM 40, YY=00>RS Port on MP>11 MCI and Built-in SMDR (Not allowed, do not set)
14 Built-in SMDR (Not allowed, do not set)
NONE (Required setting)

For Outgoing SMDR Records:
CM 13, YY=06>Sta. No.>1 For outgoing SMDR by station
CM 35, YY=14>Route No.>1 For outgoing SMDR by trunk route

For Incoming SMDR Records:
CM 13, YY=05>Sta. No.>0 For incoming SMDR by station
CM 35, YY=49>Route No.>0 For incoming SMDR by trunk route
CM 08>426>0 Effective for all incoming calls
>1 Effective only for incoming calls with Account Code
CM 08>463>0 Send ANI/Caller ID to SMDR
>1 Do not send ANI/Caller ID to SMDR

For Tandem SMDR Records:
CM 08>040>0 Allow tandem SMDR records
CM 08>803>0 Incoming and Outgoing records of tandem call is sent to SMDR
>1 Only Outgoing record of tandem call is sent to SMDR

CCIS Programming:
CM 08>378>0 Provide the system with Centralized Billing for Local office
CM A7, YY-03>CCH No.>0 Distant end is a Center office
CM A7, YY=04>CCH No.>Point Code of Center office








Phillip E. Porter
Senior Systems Engineer
Telecommunication Solutions Group, Inc. (TSG)

 

[peporter]

Hold on !!!!! I sent you the wrong one. I sent it for the IPS 2000. I will find it for the 2400 and do this again.


Phillip E. Porter
Senior Systems Engineer
Telecommunication Solutions Group, Inc. (TSG)

 

[peporter]

Well I have searched and All I can find is the manuals for SMDR and the Centralized billing. You stated you already checked all the programming via the manuals. I have tried to find any tech notes or anything and found none.
Since this has been running for a period now. The only thing I would suggest is attempt to reprogram the Centrallized billing portion of the program. Making sure it is correct, since you removed it from programming. Attempt the calls and if it still does not work, then I would reset the IO card within the 2400 then attempt it again. At least you know it is putting out the records locally, which is good. So it should not be to tough to get corrected.

Phillip E. Porter
Senior Systems Engineer
Telecommunication Solutions Group, Inc. (TSG)

 

[phadobas]

Thanks Phillip,

I had my vendor out here yesterday, verified all the programming on both switches, with NTAC on the phone. They concluded that all the programming of both systems were correct, and that I should do a power-down and -up the ICS during the night when the system is not in use that much.
Before doing this however (as I don't feel very confident powering down a 13 year old switch that NEVER went down), I'll go through the MAT log of both systems for the prior few days, to see if some change may have caused this to occur...
Will let you know.

 

[peporter]

Your the man. I have seen some issues with the ICS and even the newer IPX's. Haveing features lockup once and awhile. SMDR, PRI's, Paging access across a network. Just off the wall stuff. But low and behold a reboot fixes it. Even thought it runs for ever, (NEC is a great product) it still could use a reboot periodically. I try to shut the one down at the VA Hospital here in Lousivlle about 1 or 2 times per year. It had ran error free for 5 years now and (knock on wood) never had a issue with it comeing back up. So good luck and sorry I could not find that info earlier for you.

Phillip E. Porter
Senior Systems Engineer
Telecommunication Solutions Group, Inc. (TSG)

 

[3racefan3]

Found this in a SMDR manual.

Trouble Shooting Tip:
If SMDR stops being sent from a remote node across CCIS, but will dump out a Serial Port
from that remote node; in the remote node change the point code in ASYD Index 182 to
some other location, then immediately change it back.

 

[phadobas]

SOLVED!!!
After two weeks of switching my smdr recording connection back and forth between the two pbx-s, I got fed up with it. Although I've done this many times, I cleared and re-programmed ASYD sys=1 index 180-183 on both systems, and lo and behold it started working. Wow. Wonder what happened...
Then also set ASYD sys=1 index=288 on the 2400 to 0, just like the 3 different manuals (ICS - IMX - 8500) say.

Have been working ever since, like nothing happened.

Thanks for all the input!

 

[peporter]

Great Job!!

Phillip E. Porter
Senior Systems Engineer
Telecommunication Solutions Group, Inc. (TSG)

 

[OzzieGeorge]

Think this info is worth a star!

 

[phadobas]

The story actually did not end there...
A few days after my last post, when everything was working fine, the problem re-occured. And I could not get it working again. So again, I kept switching my accounting computer connection between the two systems.

THEN, the SV8500 stopped sending or buffering its own SMDR all together. So, now I was actually losing call records. I couldn't have that.

So, came a late night, when I decided to reboot the SV8500. PCPro Tools -> restart both sides (redundant system).

The pbx came up, but only partially. All the TDM ports (phones, trunks, etc) and functionality worked fine, but none of the IP phones worked. The 'ONLINE' light on the server was blinking, indicating that the 'SP did not start'. I believe it means that the SIP side of the system was down, didn't restart.

But, lo and behold, I got my SMDR over CCIS working again 100%!

Now I just needed my 80 SIP phones (DT700 series) up. The only way I could get the SIP side of the server going again was by un-programming all the Protocol Handlers (MPH, PHI, PHD), and re-program them. I programmed them in a different PIM, and I don't know if that's significant or not, but once that was done, the SIP server finally came up.

HOWEVER, still, none of the IP phones would register. They were asking for a password, and when I put that in, they wouldn't take it, they would just ask it again.

I ended up having to use MA4000 to unprogram and re-program all of those stations, after which, they would finally take the password and come up (took me 2 days of programming).

I had my vendor here, who got on the phone with NTAC. Here is what was diagnosed:

At the time of the install, about a year ago, when most of the basic programming was already done, something was not working right. It was either MA4000, or OAI on the OW5000 server. The installer, after consulting with NTAC, decided that a Fusion license was needed to be added to the system to make that thing work.

But the catch is that when you add the Fusion license, now you have to program stuff into the Network Data Memory instead of the local memory. Many of the commands have two versions: one ending with 'L' for local, and one ending with 'N' for network. Example: AFRSL and AFRSN, or AOPRL and AOPRN. Same goes for the commands that set up the protocol handlers. So, when the install was done, everything was programmed with the 'L' commands, and then the Fusion was added on afterwards. This made the SIP server unreliable.

As of today, about 5 days later, I still have the handlers programmed with the 'L' commands, and NTAC told me to re-program them when traffic is low (as it will kill the SIP phones for as long as the programming takes). And once again, SMDR over CCIS is not working $%$%*&%@ arrrggghh.

So, once I have the protocol handlers re-programmed with the 'N' commands, I'll let you guys know how it went.