9100 unstable Netlink, System two dropping and not rebooting during scheduled reboots. 1

[noahg350]

Hello all,

I have a 300 user 9100 bridge system with two towers, netinked together. My system is working great, up until the moment it is not. My main symptom is the primary system, ID1 locks up randomly and pull the whole system down with it. ID1 then will not shut down with power removed and switched off, (backup battery's). I read a post here today: "Netlink disconnection problem in SV9100"

https://www.tek-tips.com/viewthread.cfm?qid=1814030

and this sounds just like what is happening to me. I went ahead and made the changes in 51-06 and have yet to have a chance to test yet. Both systems are connected with fiber so I am hoping my latency is ok. Traffic or IP conflicts interrupting the netlink to ID2 might be a problem, whats a place to start to trouble shoot this?

I am also having trouble with my SIP trunks not closing and therefore they get "consumed" over the course of a day and fill up, the daily restarts (90-09) help this but it would be great to get to the bottom of it as well.

I am working in tandem with NEC tech support on this but progresses and getting my point across is not always easy.

Thanks

 

[CoralTech]

Something on the network, if the systems are working ok, is probably causing a DOS attack on something. You have wireshark on both systems to isolate this. Also, look at your alarms because they will usually show something that is causing this.

 

[noahg350]

CoralTech,

You are correct, a packet capture revealed multiple ARP probes attacking and confusing and locking everything up. Working now to identify and remove the sources. Thanks!

 

[CoralTech]

Take care because Cisco switches with CDP on cause all kinds of hell.

 

[DoubleDriveLine]

I had a customer who had a company install a network monitoring software package. The dam thing locked up the main SV9300 CPU at least once a week. Then I moved the main to the alternate site (planned move) and the company tamed the beast by not monitoring the voice subnet. That helped the lockup but was now breaking MCI so I had to reboot the PBX to fix it daily. Finally they shut it off and the problems went away.

The core issue I think was they are running this software in the same VMWare server the UM8700 ran in. It was hogging all the bandwidth on the network interface. That is just a guess but our Network team suggested the it was that.

This also happened at a Public School I know of in Western Ohio. They bought the Big full blown version of Solar Winds and it crippled everything on the network. They never could get it tamed down to where they could use it and had to abandon the purchase. Bigg $$$$ down the drain.





There are 10 kinds of people in the World.

Those that understand Binary and those that don't.

 

[noahg350]

Not out of the woods yet. The site I am at has lost track of the firewall and how to access it so I cannot take care of the issues via whitelist or blacklist. The two devices that popped up in wireshark cannot be found physically yet either, one may be an employs laptop network card. A new fire wall is a few weeks away so I may be left to play the reset game for a bit.

Question: I had a scheduled daily reset on the system, (90-09) and I found that my #2 system would never come back online after the restart. I removed the 1am restart yesterday and this morning is smoother so far. Im sure this is a network issue but it is an odd symptom.

I am also getting a "denial of service" alarm which I am being told means the system is being attacked for the purpose of locking up the VOIP card.

Maybe the resolve will come when a working firewall is installed. I believe my system is fine, just the environment it is in is not being kind to it.

Thanks