Slow Network Performance with ML350's

[Dredd123]

Hi,

I have two HP ML350 servers, as follows:

"S1" is an ML350 G4 with 3GHz CPU, 2GB RAM, Ultra320 15K disks in RAID 5 and built in gigabit NIC running SBS 2003 Premium with SP2.

"S2" is a brand new ML350 G5 with quad core CPU, 2GB RAM, SAS 10K disks in RAID 5 and built in gigabit NIC running Windows Server 2003 Standard R2 SP2.

I have been doing some testing on the network and the performance is nowhere near what I would hope for and expect it to be. I used the following configurations and tests:

CONFIG1: Both servers connected to unbranded 100Mbps switch at 100Mbps

CONFIG2: Both servers connected together with a cross-over cable at 1Gbps

CONFIG3: Both servers connected to a brand new Netgear 1Gbps switch at 1Gbps

TEST1: From the console of S1, use Windows Explorer to copy an i386 folder from S2 to S1.

TEST2: From the console of S2, use Windows Explorer to copy an i386 folder from S1 to S2.


CONFIG1 TEST1 throughput = 26 Mbps
CONFIG1 TEST2 throughput = 46 Mbps

CONFIG2 TEST1 throughput = 29 Mbps
CONFIG2 TEST2 throughput = 67 Mbps

CONFIG3 TEST1 throughput = 42Mbps
CONFIG3 TEST2 throughput = 72Mbps

The tests were performed after cold boots on both servers, with no other devices connected to the network. TCP offload engine is disabled on S2. Both NIC's and switch ports were set to "auto" for speed and duplex. With the gigabit switch and the crossover cable, both NIC's reported that they were running at 1Gbps.

I can live with the performance on the 100Mbps switch, but I can't understand why increasing the speed of the network to 1Gbps (a tenfold theoretical increase) results in a miserable performance increase.

Can anyone suggest where to start troubleshooting this?

Thanks,
Dave.

 

[ntinlin]

A lot of it will depend on the engine in the Netgear switch. Might not actually have the CPU capacity to really support 1Gbps despite its label.

Windows Explorer also puts a lot of overhead into any big file copies. Try robocopy or simple xcopy and compare the results.

Also auto speed and duplex is usually a bad idea on any server. I always force my servers to the maximum speed that the NIC and switch mutually support. It makes one less thing to worry about.
Some Cisco switches especially don't like auto settings, they end up doing stupid things like 100Mb half duplex.

Neill

 

[Dredd123]

Hi ntinlin,

Thanks for your reply. Mostly agree with what you've said (apart from the auto settings point; different people have different realworld experiences in this area) and will try robocopy/xcopy and compare. However, I ruled out the switch by using a cross over cable (and forcing the NIC's to 1Gbps although I didn't make that point in my original post) and the performance was still not what I would expect.

What sort of throughput are you typically getting with 1Gbps servers on 1Gbps switches?

Thanks,
Dave.

 

[ntinlin]

No idea. Not using 1Gbps at this site except for switch interlinks and I've never done any profiling at the other sites. :-(

Neill

 

[ntinlin]

No idea. Not using 1Gbps at this site except for switch interlinks and I've never done any profiling at the other sites. :-(

Another thing to check would be the HP NIC driver versions.

Neill

 

[TheLad]

Are you running the latest Proliant Support Pack on both servers (which will update the network driver accordingly)?

--------------------------------------
"Insert funny comment in here!"
--------------------------------------

 

[Dredd123]

Hi,

The ML350 G5 was built using Proliant Essentials 7.90. The other server is about 2 years old. Therefore I would expect neither server has the absolute latest NIC driver.

Surely HP would not release NIC drivers cobble performance so badly? This should have been picked up in testing before release?

Thanks.

 

[ntinlin]

Maybe, maybe not. Who knows how much testing they actually do.

Had a very weird issue with server teaming with latest HP NIC drivers just last week.

Teaming was installed and NIC's auto-detecting with 100Mb/Full. However since our policy is always to force the speed I forced one card to 100Mb/Full. Then all hell broke loose.
Lost remote control, drive mapping etc. connectivity to server but was able to ping it, managed to get in via iLO and was getting loads of RPC Server unavailable messages in event log, despite service being on and services that rely on it like print spooler being active.
Tried re-setting NIC to auto, no joy. Disable one card, remove cable etc. etc.
NIC's passed all diags, team passed all diags but only got it back properly on network by deleting the team and using a single card.
Very, very strange!

Neill

 

[ADB100]

Also auto speed and duplex is usually a bad idea on any server. I always force my servers to the maximum speed that the NIC and switch mutually support. It makes one less thing to worry about.
Some Cisco switches especially don't like auto settings, they end up doing stupid things like 100Mb half duplex.

10 years ago maybe. Anything made in the last five years leave them at Auto, you are asking for trouble hard-coding things.....

Andy