RMX log

[lhiraman]

Hello, I see that our IPDA remote shelf still reboots everyday around either midnight or 1 AM. I found this in the RMX log off the IPDA system.
I know that the AP backup server is set to restore on Fridays at 11PM. Can anyone give me a clue what is going on? I also have a ticket open with Siemens.


pid=2173
2173: command: /opt/hbr/rmx/rmx_backup_restore -c start
2173: Started at: Tue Sep 24 01:04:38 CEST 2013
2173: ** RMX reachable (count=12 a 6 seconds) **
2173: Exit: 0 at: Tue Sep 24 01:06:11 CEST 2013

 

[lhiraman]

Siemens level 1 told that they needed to update the NCUI board. Let's see what happens.

 

[lhiraman]

Just an update. The told me they update the software on the NCUI card because what I was telling was a known bug. I asked them to not to close the ticket. Let's see what happens.

 

[donb01]

I have been following this because I'm always interested in how things tick. I wanted to answer the photo question.

In order to post a photo to this site you must first save the photo somewhere that you can access it with a URL - like dropbox, cubby, skydrive, etc (any other file sharing site). When you post the photo there it will give you a public link to the file. When you create your post here click the little picture icon and give it the URL of the picture. It will then appear in the post (or at least be linked).

This is similar to providing links to PDF files and other documentation saved out in the cloud somewhere and linking to them from your posts.

 

[sbcsu]

FYI
The bottom RJ45 LAN connection LAN1 is always used.
If you have a PPPOE connection to the main site you can use LAN2
All above except for Version 6 where you can use LAN2 as a fallback.

If you have the latest Loadware files installed then it should have got the upgrade that you are talking about ?

 

[lhiraman]

Siemens said that they have to dispatch a local tech. They came up with these errors:

F5915 M4 N4784 NO ACT BPA LTUC PAYLOAD CONNECTION 13-10-07 13:23:13
ALARM CLASS:CENTRAL:033
** :LTG1 :LTU17:006: 0 : 0 Q2324-X NCUI4/1 BST:01 PLS:-20
FORMAT:43
REASON:00H IP BAD QUALITY
PARTY A (PEN): 1-17-1-23-1
PARTY B (PEN): 1-1-1-1-23
SOURCE IP ADDRESS: 192.168.52.15
DESTINATION IP ADDRESS: 192.6.35.107
NPCI: 1140
000000DBFFFFFFFF00000000FFFFFFFF 00000000FFFFFFFF00000000FFFFFFFF


and

From today at 00:32 I see BAD IP CONNECTIVITY:

A9001 M4 N5462 NO ACT BPA NMCALARM MAJOR ALARM ON 13-10-08 00:32:05
ALARM CLASS:CENTRAL:033
ALARM NAME:BAD IP CONNECTIVITY
FORMAT:2C

What does bad IP connectivity mean?

 

[Iamnothere]

BAD IP Connectivity" is an error resulting from your IPDA shelf having the parameter "PLCHECK = YES" in AMO UCSU. PLCHECK = Payload Check. Payload = Voice. Short explanation: the 4K has determined that the customer's network was not capable of supporting a voice call due to severe packet loss or delay. Long explanation: Whenever a call is connected between your IPDA and the Host, you are using the IPDA Speech Path, meaning that your voice is traveling across the customer's LAN/WAN. Your voice has been converted from TDM to IP packets, using a certain CODEC, such as G711. This PLCHECK parameter enables a "voice payload check", which is a means of determining the packet loss and delay of calls (the performance of the customer's network) on that shelf's IPDA Speech Path. If the system determines that the packet loss and/or delay on the IPDA Speech Path is severe, the HG3570 (STMI at Host) that was involved in the failed test is automatically shutdown. The theory: if packet loss and/or delay exceeds the "High" settings in AMO SIPCO, a normal voice call would be unintelligible because the quality of the connection would not be able to support the transmission of voice packets. Immediately upon shutting down the HG3570, the NCUI begins testing by pinging that out-of-service STMI/HG3570 using port 4001. When the packet loss and delay reaches an acceptable Low setting (AMO SIPCO), the out-of-service HG3570 is enabled for calls.

If the HG3570 does not pass the testing within a certain time frame (AMO SIPCO), the HG3570 is placed back into service, and if during calls the voice quality continues to be determined as "poor", then this entire process repeats itself.

Personally I do not enable this setting. If your network has no Quality of Service, or no dedicated Voice bandwidth, then the IPDA Speech Path will be susceptible to network traffic and other factors, and enabling PLCHECK will just remind you of that fact very frequently. In my opinion, PLCHECK should only be used on a high quality network, where packet loss and delay is rare. Also, if the Host has only one HG3570 board, if poor IP quality causes this single HG3570 to be removed from service, the system will enable Payload Survivability mode (if configured) for that IPDA shelf, where the system converts normal station to station calls from IPDA to Host (or vice-versa) to Public calls utilizing trunks at the IPDA and Host. If Payload Survivability is not configured, then callers will hear all-trunks-busy signal during the time that the single HG3570 is out of service, because there are no trunks to support the call, and the customer's network is not currently capable of supporting a voice call.

There are too many red flags involved in your IPDA, APE, and Host setup to guess where the problem is. And now, poor network quality is added to that list! And Loadware, too!!

 

[lhiraman]

WOW, thank you for the explanation. HOw do I check the PLCHECK setting?

 

[Iamnothere]

You can DIS-UCSU:AP,1,17;

But that AMO is a bit difficult to read, since the parameters are shown in one box, and the "values" are shown in another box just below the parameters. Perhaps the BEST way to see it is to REGEN-UCSU:AP,1,17;
However, you might first want to perform CHA-FUNCT:REGIN=YES,REGKEY=YES;
With that AMO FUNCT change, you will see the parameter names inside the AMO UCSU REGEN. Look for "PLCHECK=YES".
I highly recommend that you set it to NO (use CHA-UCSU:AP,1,17.....). Your IPDA/APE is already distressing everyone, including us readers out here on the internet in our pajamas.
You do not need another problem like BAD IP Connectivity. This may be an indication that your network may not be as robust as you think it is. I am very surprised that you have not experienced that error previously.
Perhaps Siemens made that AMO change to see if your network was behaving. Remember I explained previously that it takes only 64 seconds of continuous network outage to trigger the AP-E to take control of an IPDA shelf (using the AMO SIPCO -> TIMING default values.)

If/when the Siemens tech visits your site, I recommend that you ask the technician to test the AP-E switchover using EXE-APESU. This command will cause your IPDA shelf 17 to go down, switchover to control by the AP-E. Then the technician should switch it back to control by the Host, which also takes the IPDA shelf down. If I were you, I would not allow the technician to leave until it is working properly!

 

[lhiraman]

Well, Everyone here is an update. They can not find the issue! Nothing wrong on the network, they swapped out the NCUI card, switch into AP and back. They can't find anything. Now they are telling me that the host can't ping the CC-AP; I had them use the same IP/subnet/etc on a configured laptop and the ping worked.

Is it possible that the CC-AP card is disabled? If so, how can I check? I asked them and they said it automatically is enabled.

 

[Iamnothere]

If the system switched into APE mode, and then back to Host control, then why would a "ping" be an issue?

Perhaps I need a reminder of what the problem actually is now. I thought I remembered that the Access Point Emergency server was not working. But the tech proved that it IS working.
Are we looking for the reason that the shelf is re-booting, or the APE not working, or BAD IP Connectivity?
Please refresh our memory.

 

[donb01]

I believe the shelf was rebooting every day at a certain time if I remember correctly - something like 4AM. Maintenance is usually at 2AM, so that's a couple hours after - probably far enough that it has nothing to do with it.

 

[lhiraman]

Hello...I escalated the ticket with Siemens. Yes the switch restarts every night at 12:30 AM, and every Wednesday at 4:30 PM. I thought that it could be a schedule running or something, but Siemens assured me that there isn't a schedule. They are telling me that on the switch IPDA CC Access point the host can't communicate with it.

The strange thing is nothing happens on Saturday or Sunday. They ran a network test, had us capture packets when the system goes down at night and it does not show any drops or time-outs.

On our network, we see packet lost around 5-6 AM went our backups take off.

Yesterday around 2:39, I received a call saying that the switch when down for 2 minutes. They told me that they need to replace the power supply...WTF...I going to wait until next week to see what happens and then escalate again!

 

[lhiraman]

Everything was pushed up to a level 3 engineering and within 1 hour, he found the problem. Turned out to be firewall dropping TCP=AF31 traffic.

So far no reboots, restarts, etc.