Serious Network Problem with HP2524

[DanielmShelly]

I have a serious problem on 2 separate networks!!!

We started seeing entire network devices become unresponsive for no apparent reason, we replaced virtually all the equipment and still it persists.

Then it started to appear at another of our office networks, configured in a very similar way.

After weeks and weeks of failing network, the only common thread seems to be invalid ARP entries, we've seen too many 00-00-00-00-00 for the default gateway(s)

Equipment is HP 2524 Switches (which I've replaced with Cisco 2950's at one office)

We have Netscreen Firewalls in both offices

My question is, is there any known issue with HP2524 switches are prone to issues (that's putting it as politely as I can)
The reason I ask about the HP's is that I've also experienced the MAC Address of 2 HP switches also reporting 00-00-00-00-00 and...needless to say I sent those switches back!!

By the way just to confuse matters I've seen 2 HP's at the 2 sites BOTH LOSE THEIR CONFIGURATION....can this be caused by buffer overflow

Please no-one mention upgrading firmware....that was the first thing I did!!!

COmplicated I know!!!!!!

GRRRRR..I'd appreciate any help!

 

[ralpheb]

Hello DanielmShelly -

I might be able to comment if I had a bit more detail. For example, what do you mean by "we seen too many 00-00-00-00-00 for the default gateway"? This is too many bytes for an IP address and too few for an Ethernet address. Are you saying that the Switch is sending packets with this destination address to the default gateway, or that the Switch itself is ARPing for the default gateway, but using this addess as the MAC/IP SRC/DST (?) address?

When a buffer overflow (how did you determine that this is what happened?) caused a loss of configuration, did the Switch reboot itself? If so, then there are probably crash files that HP Support can use to diagnose the situation.

Sorry that I'm offering you more questions than answers!

Regards,
Ralph

 

[DanielmShelly]

Sorry I got tired of typing 00-00's

That is the MAC Address that the server shows in it's arp cache for it's default gateway (the netscreen firewall)
This is local so the server itself is sending ARP requests onto the switch and something is replying with that address of zero's

As for the buffer overflow, we're making a leap as we've seen log messages saying Out of Packet buffers and when the switch fails this log is cleared so I'm merely asking if that is a possible cause?

I hope that makes it clearer!!!

Dan

 

[BAsh12]

Sounds wierd.

I have seen similar situations when broadcast storms are running. the buffers overflow and the mac table disappears. The last time was a virus, the time before a DHCP server had a fault and kept disappearing, that caused all hosts to broadcast for the server (a storm). The later could make sense as the problem is not confined to one broadcast domain (assuming you have routers for interbranch). max broadcast rule of thumb is 5% of the traffic, NOT the bandwidth.

try a packet trace, mirror the server port to a laptop running something like etherpeak and post the results.

The Cisco switches may be handling the storms better as the CPU and memory on them is much faster.

Hope this helps.

 

[ralpheb]

Hello Dan -

OK, I understand better now.

When folks say "buffer overflow" they usually mean that software tried to read or write past the limits of some code buffer, which can cause memory corruption or crashes in some cases. This is different that exhausting the pool of packet buffers, which is a designed-for condition.

In addition to BAsh12's excellent suggestion, I would like to point out that there are some excellent freeware virus- and worm-detection tools of which Snort (

http://www.snort.org)

is a good example. Snort runs under various computer OSs (for example, Windows or linux) and looks for known virus and worm 'signatures'. Unless you spend a lot of time educating yourself about current virus and worm signatures it is hard to do as good as job as Snort can do.

Is there any indication that the Switch is rebooting itself during these episodes?

Ralph

 

[BAsh12]

If you are in the uk, give me a shout. I am fairly sure I can fix it. Never been beaten, and never will.