NEC SV9100 IPL DSP is sporadic not working after upgrade to12.10.50

[timqa]

After upgrade all worked but after several days, IPL DSP (84-26) is no longer ping-able and IP phones do not work. If you change the IP, everything works but stops again several days later. You can change it back to the original IP and it works again.

 

[CoralTech]

Pull up your alarms. Bet you anything this is a network issue. Easiest way to test is if it happens again simply pull the ethernet cable and stick it into a laptop that is setup with a static IP and see if it pings. Chances are something is IP conflict or DOS attacks.

 

[timqa]

I was thinking that this is a network issue at first but not sure now. The IP address in 84-26 does not have a physical port like the CPU (10-12). All of the digital phones stay working when this happens. They have been using the same IP address for 84-26 for many years. it was only after the upgrade that this started happening. The IP phones are only losing audio when this happens. Nothing else seems to take the IP since you can't ping it. If you change the IP then change it back, they start working again.

I have numerous other systems on this same version (12.10.50) and they don't have this issue. I saw an unrelated comment about someone loading the upgrade again. Not sure if that would help.

 

[CoralTech]

Next time it happens you need to see where the issue is. If you can't ping the address you have to do simple network diagnosis. Just because you did an upgrade doesn't necessarily dictate a phone system issue. Without trouble shooting you will just be pissing in the wind. Just do the test and get it dialed down. Changing the IP just forces the IP back into service. Pulling the alarm data will go far into seeing what is going on. I've seen DOS attacks do precisely this. Everything from something pinging it to Cisco CDP junk.

 

[timqa]

Some more info on this. May not be relevant but want to be thorough.
This was an SV8100 that was migrated. At least a couple of years since the migration with no issues. After the upgrade to 12.10.50 from 8.00.65 approx. a week later the IP phones lose audio and no issue with digital phones (as expected). IP address in 84-26 (172.30.6.252) no longer responds to pings. Change address to 172.30.6.253 and everything works. Next time it goes down you can change it back to the original address and all works again.

Went on site today 12/20/23. Alarm log shows nothing since 11/17 and the only thing it does show is SMDR buffer full and previous planned restarts. The customer has also tried changing the address and immediately changing it back and it works again. 84-26 can have several different addresses over a period of time and they all have the same issue.

I don't really have direct access to the customer's network but if you have a specific test we can run, I would be glad to try it. 84-26 is an internal address and has no cable connection. I try not to point fingers at a cause until I know but I don't want to rule anything out either.

 

[CoralTech]

Did you run a wireshark? It can be run right off the system. I've never seen a full SMDR buffer bring a system down but, might as well shut that off if you are not using it. I mean you could have a bad IPLE card but I dunno. Usually that is a work/no work situation. Still sounds like a DOS scenario and the port is shutting down. Wireshark....

 

[timqa]

I agree about turning off SMDR. I don't want extra variables. I can run WireShark and it may show a constant issue but if it is one event that is causing it, it would have to be running when it happens (which is about every seven days). I reloaded the last upgrade and if it occurs again I will try the WireShark. Not sure why it is only causing issues with 84-26 since the IP address is changing. I will update this. Thank you for the info.

 

[timqa]

I have turned off SMDR and I noticed that Automatic System Reset (90-09) was set to:
Month-February
Day-2
Hour-23
Not sure if this has anything to do with it but since this issue was happening every Saturday at approx. 11 PM, I did not want to chance it. We will see next weekend if it happens again. I have been looking at the Wireshark but not sure what is causing the issue. I see a lot of errors stating "Bad UDP length".

 

[CoralTech]

Bad UDP length usually means Jumbo Packets. Something not configured right?

 

[timqa]

Not sure where to start to see what is not configured right? Does that look like it could be the cause of this issue?

 

[DoubleDizzle]

Hello timqa,

Just a thought....

You could deploy a test laptop running wireshark, install a port mirroring switch and a set up a continous "Ring Buffer" file.

When the event is reported, you can go back to the timestampped file and analyze it in wireshark to see what happened at moment it happened.

This is not possible with the packet capture tool built into the NEC because the capture maxes out at 300 seconds.

 

[timqa]

I have a laptop set up with a port mirroring switch and running wireshark. We have pings going to show when it stops replying. Not sure what a "ring buffer" file is. Could you explain when you have time?

 

[DoubleDizzle]

Hello timqa,

I have provided a document attached to this post which outlines setting up the ring buffer file.

If you can set up remote access to your test laptop, you can view the captured files once the incident is reported.

 

[DoubleDizzle]

Hello timqa, if you need to add wireshark to the ring buffer folder path, see attached document.

 

[timqa]

I think I found it in the software. That is where you split up the trace into smaller segments to make it easier to search and also small enough to email correct?

 

[DoubleDizzle]

That is correct.