IPDA and APE both drop out at the same time--network issue?

[kevin906]

See the attached PDF.
Customer IPDA shelf 18 and CC-AP 18 showing short duration outages of less than one minute.
My thinking is since these are two different pieces of hardware this must be a network issue..
The NCUI and AP logs that I can pull don’t indicate power loss to the shelf.
Customer claims this has to be Siemens problem.
They also have another shelf 17 with no CC-AP that is not having any problems. It is on another subnet.

Is there anything other than a network condition that would cause both the NCUI and CC-AP to both go offline at the
same time?? Customer claims no network issues and this keeps happening with outages of less than one minute.
I have pulled the error logs from the NCUI card. Logged into the AP processor, I don’t see any signs of power loss reloads
on either the NCUI card or AP. There is also a shelf 17 which is not affected during these outages. They are on a different
network address and subnet.
So the STMI back at the main PBX is not the culprit either.
There are no indications of power losses at the IPDA-APE power source.

This is not happening due to AP backups. Those are programmed to happen once a week.
These outages are daily at intermittent times of less than one minute.
Customer complains of cutoff during conversation at the same time frames.

 

[Moriendi]

If you lose the CCAP and IPDA at the same timne, it's surely network. What did the event log say from the NCUI for that time period? Would also be worth looking in hista for Bad IP messages, see how much packet loss there is.

 

[donb01]

I will be interested to hear if your HISTA log around the time of the event shows any references to Layer 1 signalling failures. I get a good bit of this but don't have the network knowledge to try to figure it out, and the network people insist nothing is wrong with the network because SoloarWinds says so. I'm all like "how do you know you even have SolarWinds set up to see those kinds of errors?" - I think all they are looking at are the big picture stats and not the small picture stats. I have one site where the IP Quality seerors were so bad (whether originating or answering) that we went ahead and effectively disabled the statistic. The network people just say things like "that's odd - we don't SEE any errors on the network", and I'm like PCs could care less, but phones have no tolerance for that crap....

One thing I will say is that someone should check the switch ports that anything I will refer to as "VOIP Backend" (STMI, NCUI, APE, Processor network feeds) has all these things removed from the port config:
LLDP transmit and receive
Remove network-policy
Lldp med tlv-select

Ports that have phones on them should have that stuff left alone - especially if you're using a DLS server for config stuff.

Don Bruechert, Voice Comm Analyst II
CareTech Solutions @ Holy Family Memorial
Manitowoc, WI, USA

 

[kevin906]

Thanks for the replies. Attaching the hista file and the error log from the NCUI card. Not an excess of payload errors for this location. There is one CC-AP dropout related to the AP backup on 5-10 at 0551 which is the APE reboot so that is not related. The other NCUI and AP dropouts are the issue. At the same time for some of these outages you will see STMI HFA phones dropping and re-registering. These phones are not in the same building or network. There has to be something in common but the IT people are adamant it's a PBX issue...

 

[kevin906]

Hista file on this upload

 

[Moriendi]

There's packet loss reported in both the hista and evtlog, sometimes as much as 20%. You will only see that when there are calls in progress, that is what is reporting the packet loss. If there are no calls you won't see a bad IP message but the shelf will still crash.

Mirror port configured on both 4K side and IPDA side would give you something to compare in Wireshark but it will only tell you what you already know - the network sucks. But if they are adamant it's PBX not network you'll need to prove it with an external wireshark trace.

Check QoS config, could be something as simple as that.

 

[hamncheese]

Make sure no one has assigned the static IP's for your ipda and ape to something else.

 

[sbcsu]

Yes, that is the most likely culprit.
Turn off your equipment and ping the address range to see if other equipment is using any of the fixed IP's assigned to the 4K