10Gb NetXen Adapter with 1Gb Bandwith

[hiasor]

Hello All!

I tested a 10Gb NetXen Adapter (HP NC510F PCIe 10 Gigabit) on DL785 with a NetApp FAS 6080 10Gb Adapter -> direct Attached over Fibre (server to server)
But i get a maximum of 140Mb/s trough the Adapter. that´s too slow.
I have testet different RHEL Kernels, different adapter drivers. always the same result (+- 20Mb/s)

Has anyone expirience with 10Gb Adapters direct attached over fibre on RHEL?

best regards,
hiasor

 

[chgwhat]

I think you have to look at other settings, you did not say, what kind of disk drives are in the NAS, what are your raid settings, stripe size, segement size, number of disks, are you using NFS, or ISCSI, if using nfs what is your block size, if using ISCSI what is your frame size, Can you monitor the link, the NAS, or Redhat server, and see where the bottle neck is.

Tony ... aka chgwhat

When in doubt,,, Power out...

 

[hiasor]

Hi Tony,
we only want to test nfs. so we tried different mount settings (rw,hard,intr,fg,tcp,rsize,wsize,jumboframes, kernel parameter, etc.)
on NetApp-side we have also tried different settings -> no impact
NetApp side is idling.
may be its a comminication problem, because it´s direct attached.

 

[hiasor]

ANY IDEAS???

 

[IRudebwoy]

If you're writing from disk to disk over the 10Gb link you're seeing the I/O limit on one or both devices not the ethernet link maximum.

Example: When writing from my laptop HDD to my server array over a 1Gb link the transfer rate I see is the maximum I/O of my laptop's HDD (28MB/sec). When doing transfers from a 100Mb link the transfer rate maxes out at the link speed (11.5MB/sec due to the protocol overhead).

Clear as mud?

 

[hiasor]

I write from NFS-Share to NFS-Share (single and multithreaded)
also only over IP. (single and multithreaded)
It should be a minimum of 600MB/sec.

 

[chgwhat]

That was part of my initial post.
If you are reading / writing to a NAS that is raid6 on 7 SATA disks, lets say each disk has a sustained write of 28MB/sec, you should expect about 6 * 28 or 168 MB/sec. What is your disk configuration on the NAS.

Tony ... aka chgwhat

When in doubt,,, Power out...

 

[hiasor]

@Tony:
it´s only one Diskpool with 72 disks. The disk´s aren´t the limiter.

 

[hiasor]

@Tony:
and as I wrote before, we testet only over IP - no disks involved - the same result

 

[IRudebwoy]

What happens on the RedHat side? CPU usage stats would be a start.

Are you using NFS over UDP or TCP? If it's TCP it might be the verification of the packets that's causing the bottleneck.

 

[Annihilannic]

So when you only tested over IP, what data were you using for the test, and where was the data coming from and going to?

Annihilannic.

 

[hiasor]

we testet only over ip and with nfs. we also have tried tcp and udp. and we testet with multi-threads so the cpu usage isn´t the limiter.
has anyone expirience with 10Gb over Fibre. because it looks like 1Gb half duplex. we see sending pakets and then recive.

kB/s kB/s
out write
29583 0
38527 0
44404 0
65660 0
86231 318852
84163 118932
85713 111152
30690 111888
11640 103748
35746 78848
37693 74600
71208 36
80892 0
83681 0
83268 0
59183 0
3320 0
19618 0
37851 0
45652 0

 

[Annihilannic]

You still haven't described what test data you are using? NFS is a filesystem, so you can only test it by reading and writing files. What files are you reading and writing? Where does the data in those files come from?

Annihilannic.

 

[IRudebwoy]

Okay, at least try [tt]ethtool[/tt] to determine the negotiated link speed.

Remember, you asked this community for help. You can always pay for it. It would be prudent to answer some of our questions amid your frustration. Like Annihilannic I would like to know how you are testing a file system, especially without involving disks. I would like to know the process you're using. I do something similar by using [tt]dd if=/dev/zero of=/remotefs/file[/tt].

BTW, 10GbE host adapters require a connection to a switch to attain 10Gb speeds else they default to 1Gb half-duplex. Read up on your system docs and standards.

L8