ProCurve compatibility issue with Etherwan?

[JBruyet]

Hey all,

I have a fiber segment that’s slow a lot more often than it’s fast. Here’s the setup: At this end I have a 4000M that connects to an Etherwan Xpresso switch (from the vendor) and then the fiber). Then at the other end of the fiber there’s another Etherwan Xpresso and then a 2524. The fiber segment is 10Mbps, but about 75% of the time it runs at about 150Kbps. There are no errors in the logs of either of my switches and no errors in the logs for the switches from the vendor. I’ve run Ethereal and I don’t see anything that would indicate a temporary broadcast storm (is there such a thing?) since the problem is intermittent.

I’ve made many phone calls and had techs over trouble-shooting and I’m hearing that it’s a compatibility issue with my ProCurves and the “other stuff” they’re connected to. At first I doubted this but I “borrowed” two Allied Telesyn switches to use in place of the ProCurves and things are zipping along with no slowdowns.

Has anyone else heard of this happening? After investing in the ProCurves (and some GB modules for a different long run that needs high bandwidth) I don’t really want to buy more switches, but I’m starting to think that’s the direction I need to go. If anyone has any ideas or suggestions I’m open to just about anything.

Thanks,

Joe Brouillette

 

[ralpheb]

Hello Joe -

You don't say what traffic level your end nodes are capable of generating, nor how you measured the 150kbps (in other words, from where to where did you measure this?), so I don't pretend to have a firm understanding of your situation.

I will say that if your end nodes are trying to generate a lot more than 150kbps, then the problem should show up in an error counter somewhere (reference:

http://www.hp.com/rnd/library/troubleshoot_lan.htm).

You say that you studied the 'logs', but it isn't clear whether you made a careful examination of the Ethernet and drop counters, end-to-end, including the PCs. Did you do that? Usually, network slowdowns like this are due to dropped packets, and dropped packets usually are reported as Tx or Rx Drops, or as Ethernet errors (excessive collisions, CRC errors, etc.).

Good luck,
Ralph

 

[JBruyet]

Hi Ralpheb,

I found the counters and discovered that we're dropping some packets. Here's a sample of three days of data:

Link Status : Up

Bytes Rx : 149,689,033
Unicast Rx : 121,046
Bcast/Mcast Rx : 1248

FCS Rx : 533
Alignment Rx : 3
Runts Rx : 0
Giants Rx : 3
Total Rx Errors : 539

This data was from my "monitoring port" but none of my other ports showed any errors. This switch is connected to an ethernet port on my fiber vendor's switch, and on the other end of the fiber line things are clean. I was hoping that there would be an active port (with a real live user connected to it) with errors on it so I could narrow the search down some but such is not the case. Any ideas on where to look according to this?

Thanks,

Joe Brouillette

 

[ralpheb]

Hi Joe -

So, you are telling me that these FCS and Giant errors are on a port that you are using for monitoring and which is connected to a PC (running LAN analysis software) or a LAN analyzer, right? If so, then these errors are unrelated to the performance problem.

Are the Drop counters zero on all of the ports along the communications path that you are troubleshooting?

You should have a look in the event logs of the ProCurve switches and Xpresso switches to see whether there are any non-Informational entries. Do you see any "W" (Warnings)?

Ralph

 

[JBruyet]

Hi Ralph,

I've shutdown my packet sniffing software.

Yes, all the drop counters are zero on the ports along the network path. That's what is so confusing and frustrating.

I get a random W in the event log now and then for random ports and that's it. Please see the following post I put in another forum:

I’m having a problem with the connection between a Novell server and my ProCurve 4000M. I’ve been trying to find a cause for a slow fiber link and discovered that packets are getting dropped between the server (a Compaq Proliant 1600) and the switch. This connection goes from server to wall jack to patch panel to switch—it doesn’t use the fiber.

I started a Novell ping from the server to the switch and while I was back watching the counter I saw the connection fade (several dropped packets) and then drop out completely. After about 15 seconds the connection slowly came back up (fewer and fewer dropped packets). After about two minutes this happened again while I was sitting and observing. One other thing I noticed is that at least once every 10 packets the ping ack is over 300ms, and several of those are over 600ms. The slowest ack was just over one second. I used a brand new Belkin cable and plugged the server into another wall jack and I’m getting the same results.

I checked the switch log to see what the drops are from (I was thinking CRC errors) but the switch log doesn’t show any dropped packets. Has anyone heard of any compatibility problems with older Compaq servers and newer ProCurve switches? I suppose I could buy a different brand of switch and see if that corrects the problem, but my 4000M is also using a fiber module for a gigabit run to some other servers and I don’t really have the $$$ to do any experimentation. Any help would be greatly appreciated.

Ralph, does any of this make you snap your fingers and say "Oh yeah! I know what that's from!" I'm ready to grasp at any straws as I have users getting pretty upset because of this.

Thanks,

Joe Brouillette

 

[ralpheb]

Joe -

I'm not snapping my fingers yet, but let's not give up yet, either.

I did not see your posting in the other forum. What is the Switch's "W" event log message text? That could yield a clue. Also, do your PCs offer any Ethernet error statistics? Perhaps the PC thinks that some of the Switch's packets are bad. Finally, is this problem peculiar with a particular type of NIC (in the PCs)? If so, what NIC?

Ralph

 

[RichieClarke]

Sorry I would like to help but am not clear on the problem, you say you have a problem with a slow network link running 10mb over fiber? why would you run 10mb over fiber? is it a LES10 circuit or is it private LAN connection?

Are you getting any late collisions on this line, if so you may want to try running it in full-duplex mode.

Need info on what is on either end of fiber.

Usually slow ethernet networks are an indication that somewhere you are out of spec with the ethernet/fastethernet standards, if there is no faulty hardware anywhere. Its surprising how easy it is to fall out of spec, you add a hub here and there and before you know it you have problems and you wonder why.

Anyway let me know more info and I will see if I can help.

Cheers,

Richie.

 

[ralpheb]

Hello All -

Besides out of spec equipment or broken hardware possibilities that Richie pointed out, other possible reasons for slow Ethernet performance are:

1. Dropped pkts. This is usually due to oversubscribing
a network device or link and is a network design issue.

2. Duplex mismatches. Under heavy traffic load, the
resulting CRC errors and late collisions will result
in poor performance when viewed from the perspective
of a user session or application.

Regards,
Ralph

 

[JBruyet]

Ok, now I'm thinking it's my Novell server. Here a screen shot of the ping status. So far I've lost 28 packets. My server is connected by a new cable directly to my 4000M.
+------------------------------------------------------------------------------+
¦ Ping 4.23b NetWare Loadable Module ¦
+------------------------------------------------------------------------------+
¦+----------------------------------------------------------------------------+¦
¦¦ Node Sent Received High Low Last Average Trend ¦¦
¦¦ 192.168.2.13 1,242 1,214 98% 233.5ms 0.9ms 1.1ms 4.5ms 10.6ms¦¦
¦+----------------------------------------------------------------------------+¦
¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦¦
INS=Insert Target DEL=Delete Target ENTER=Modify Target ESC=Exit

And here you can see that there are no drops or errors showing up on the 4000M port that the server is connected to. AND, I have triple-checked to make sure the port number is correct.

Link Status : Up

Bytes Rx : 2,025,057,903
Unicast Rx : 9,877,380
Bcast/Mcast Rx : 37,539
Bytes Tx : 1,328,887,948
Unicast Tx : 9,827,047
Bcast/Mcast Tx : 1,730,731

FCS Rx : 0
Alignment Rx : 0
Runts Rx : 0
Giants Rx : 0
Total Rx Errors: 0
Drops Tx : 0
Collisions Tx : 0
Late Colln Tx : 0
Excessive Colln: 0
Deferred Tx : 0

Yeah, this isn't the fiber connection but I'm wondering if it's related.

thanks,

Joe Brouillette

 

[JBruyet]

Hi Richie, sorry I missed you in my last post.

I'm running 10Mb over fiber because I can't afford 100Mb. We are running a VLAN.

We're getting 0 errors on this line, and I've already changed from half- to full- and back several times. This along with phone calls to my vendor support to make sure they are setting their equipment the same.

I have six users at the far end of the fiber, and about 45 users at this end. Local speeds are great at both ends of the connection. We just can't seem to get consistent high speed over the fiber. Now I'm going to concentrate on the Novell problem listed above because that's the only problem I can see that's leaving errors. We'll see...

Thanks,

Joe Brouillette

PS HP tech support hasn't been much help yet but they showed me how to export some diagnostic files out of my switch. I'll let you know what I find out from HP.

 

[RichieClarke]

Hi Joe,

Yeah I think the novell server could be an issue here if users at both ends are accessing it. Might be worth changing the NIC in your novell server.

But another area to check is SAP packets, I had a similar problem on a novell network a few years back, I remember we traced it back to a faulty HP Jetdirect box sending out loads of SAP advertisements.

I don't know whether you have tried this but if you haven't it may give you a clearer idea of what is going on, setup a port on your hp switch as a monitor port, install hp toptools on a machine, plug the machine into the monitor port you set up ealier and monitor the switches activity for a day or two. You will be able to see which packets are the most popular and whether the amount of packets that flow accross your network are what you would expect.

Hope this helps you.

Cheers,

Richie.

 

[RichieClarke]

Hi Joe,

Sorry about all the posts that i'm putting here but something just came to mind.

We have a 10mb fiber link to a site that's provided by our telecoms company (a leased line) and we had exactly the same trouble you are describing apart from the fact that our servers are win2k servers, anyway the line was supposed to be basically a LAN extension line, so with that in mind I went ahead and installed a HP 2524 switch at both ends and we had network connectivity. The fiber link is terminated to a device at both ends which has an ethernet port on it (RJ45 10mb) and I just connected those ports to the switches at either end. The network connectivity was definately there alright but the link was so slow that it was practically unuseable. I tried various things to try and diagnose the problem but nothing gave me a solid answer as to why I was getting this problem. The telecoms company advertise this link as a LAN extension and claimed that you could just have a switch at either end, I suppose technically they were right but the link is far to slow to use like that, we had 2mb kilostreams that were faster than this and this was supposed to be 10mb. So anyway to cut a long story short I did some research on it and it turns out that anyone who used these lines didn't put a switch on either end they routed over it. So I ordered an Ethernet module for our cisco 3660 at our end and ordered a new cisco 2611xm for the other end. I installed a domain controller at the other site along with the new 2611xm and set the link up through the routers. I tested the link speed by browsing the network and copying data between sites and guess what,,,,,it was 100 times faster. Now you may think thats an exageration but i'll give you an example,,,when we had the switches I copied a 26mb file from one site to the other and it took approx. 6-7 mins, when I had the routers put in I copied the same file and it took approx. 30 secs.

I know cash can be an issue for some companies but if your scenario is pretty much the same as mine then routing is the way forward. I paid around £1200 for the cisco 2611xm and £2-300 for the Ethernet module for the cisco 3660, a small price to pay for getting the most out of your intersite links I think.

I hope that helps you.

Cheers,

Richie.

P.S. If your scenario is nothing like mine then just ignore this post.

 

[JBruyet]

Hi Richie,

One of my vendor techs is leaning the way you mentioned concerning having switches at each end of this fiber link. His thought is that the problem has to do with compatibility issues with the vendor switch/my ProCurve switch interface.

I'm currently waiting to hear back from HP tech support concerning the 4000M/Novell server problem. I'm hoping that somehow getting this fixed will correct the fiber thing too. If not, I'll look into "borrowing" some routers from my vendor to see if that helps.

Thanks,

Joe Brouillette