Slow Backup over Gigabyte Network

[octar]

Hi,

I have two model 7026-H80's and a M80 at AIX 4.3.3 ML 11
Recently the network was upgraded to use Gigabyte

In doing so the backup times have not improved, I have performed the following tests to try to resolve but still not getting anywhere, can anyone provide advise on this?

note: I have checked with "netstat -a | grep EST" to see the ftp connections are going over the Gigabyte card.

TESTS:

7026-H80 (Development Server) 2xCPU's 3GB Mem backups 4hrs+ (62GB)

ftp command put "| dd if=/dev/zero bs=64k count=32000" /dev/null
ftp (local) 2097152000 bytes sent in 136.9 seconds (1.496e+04 Kbytes/s)
ftp (to tsm) 2097152000 bytes sent in 114.1 seconds (1.795e+04 Kbytes/s)
ftp (from tsm) 2097152000 bytes sent in 104.6 seconds (1.958e+04 Kbytes/s)


7026-M80 (Production Server) 4xCPU's 4GB mem backups 30Mins (64GB)

ftp put "| dd if=/dev/zero bs=64k count=32000" /dev/null
ftp (local) 2097152000 bytes sent in 86.69 seconds (2.362e+04 Kbytes/s)
ftp (to tsm) 2097152000 bytes sent in 89.47 seconds (2.289e+04 Kbytes/s)
ftp (from tsm) 2097152000 bytes sent in 97.79 seconds (2.094e+04 Kbytes/s)


7026-H80 (TSM Server) 4xCPU's 3GB Mem
ftp put "| dd if=/dev/zero bs=64k count=32000" /dev/null
ftp 2097152000 bytes sent in 93.63 seconds (2.187e+04 Kbytes/s)


Is this possibly the restriction on the CPU ??

thanks

 

[MoshiachNow]

Hi,

In order to get a clear picture you have too use vmstat and topas DURING The test to establish the bottleneck.

However,first I would check your switch port settings.

"Long live king Moshiach !"

http://www.7for70.com/

 

[slamcorp]

wow that is pretty slow, my recommendations would be;

/usr/sbin/no -o rfc1323=1
/usr/sbin/no -o tcp_sendspace=131072
/usr/sbin/no -o tcp_recvspace=131072
/usr/sbin/no -o udp_sendspace=65536
/usr/sbin/no -o udp_recvspace=65536

also jumbo frames but that requires switch changes usually.. i've gone back and forth on these for a long time and most of it has to do with hardware sysplanar/system board and i/o throughput;

my ftp 'findings;
h50 - 25 mb/sec
h80 - 50 mb/sec
6m1 - 80 mb/sec
6x0 - 95 mb/sec

 

[jimh719]

If your switch is not set to Jumbo Frames, then change your MTU size back from 9000 to 1500 for each of the Gigabit Ethernet cards, using SMIT-TCPIP,Further Config, Network Interface, Network Interface Drivers.

 

[MoshiachNow]

That's what I meant - the speed of 1.5 MB/Sec does not point to network params problems on tcp/udp level (they would matter if performance needed to be increased,sa,from 20 to 40 MB/sec).

It's got to be a mismatch between the card and the switch settings.

"Long live king Moshiach !"

http://www.7for70.com/

 

[octar]

STEP 1
Change: (As suggested)
/usr/sbin/no -o rfc1323=1
/usr/sbin/no -o tcp_sendspace=131072
/usr/sbin/no -o tcp_recvspace=131072
/usr/sbin/no -o udp_sendspace=65536
/usr/sbin/no -o udp_recvspace=65536

LOCAL FTP

>vmstat 5
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
2 2 855028 699 0 2 2 284 1171 0 315 2417 630 7 14 74 5
2 2 855095 616 0 1 0 0 0 0 218 3725 876 19 17 63 1
2 2 855095 614 0 0 0 0 0 0 212 9423 2274 53 42 6 0
2 2 855095 612 0 0 0 0 0 0 218 7955 2215 52 42 6 0
2 2 855095 612 0 0 0 0 0 0 218 9526 2331 53 41 6 0
2 2 855095 611 0 0 0 0 0 0 224 7932 2383 55 39 6 0
2 2 855095 579 0 0 0 0 0 0 230 9798 2339 53 41 6 0
3 2 855095 573 0 0 0 0 0 0 216 7988 2320 51 43 6 0
2 2 855095 573 0 0 0 0 0 0 213 9273 2397 52 41 7 0
2 2 855095 573 0 0 0 0 0 0 268 7568 2275 50 46 4 0
2 2 855095 571 0 0 0 0 0 0 213 9434 2268 53 42 6 0
2 2 855095 446 0 16 0 0 0 0 249 9655 2348 57 38 4 0
2 2 855095 1417 0 0 0 207 1185 0 225 10009 2356 52 42 5 0
2 2 855095 1347 0 10 0 0 0 0 230 7952 2305 51 44 5 0
3 2 855095 1336 0 0 0 0 0 0 213 9522 2294 52 42 6 0
2 2 855095 1326 0 0 0 0 0 0 221 7943 2328 54 40 6 0
3 2 855103 1268 0 3 0 0 0 0 230 9534 2316 52 41 7 0
2 2 855103 1264 0 0 0 0 0 0 231 7924 2241 51 42 7 0
2 2 855103 1264 0 0 0 0 0 0 225 9572 2280 54 41 6 0
2 2 855103 1255 0 0 0 0 0 0 222 8006 2303 55 40 6 0
2 2 855103 1251 0 0 0 0 0 0 215 12363 2299 51 43 6 0


ftp> put "| dd if=/dev/zero bs=64k count=32000" /dev/null
2097152000 bytes sent in 97.95 seconds (2.091e+04 Kbytes/s)

FTP TO TSM

>vmstat 5
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
2 2 854904 1075 0 2 2 284 1171 0 315 2418 630 7 14 74 5
1 2 854904 963 0 0 0 0 0 0 2042 4636 1709 37 34 28 2
1 2 854904 951 0 0 0 0 0 0 2097 5548 1669 31 31 37 0
1 2 854904 949 0 0 0 0 0 0 2088 3967 1662 30 34 36 0
1 2 854904 949 0 0 0 0 0 0 2087 5542 1661 32 32 35 0
1 2 854904 932 0 0 0 0 0 0 2110 3977 1648 31 38 26 4
1 2 854904 913 0 0 0 0 0 0 2072 5511 1661 33 31 36 0
2 2 855475 1119 0 1 23 205 1911 0 1904 10192 1524 46 35 18 1
2 2 855082 1598 0 0 9 0 0 0 1874 19383 1626 39 44 14 3
2 2 854905 1680 0 0 0 0 0 0 1870 13601 1573 45 37 18 1
1 2 854905 1680 0 0 0 0 0 0 2071 5683 1654 32 33 35 0
2 2 854905 1661 0 0 0 0 0 0 1979 5256 1640 42 37 21 0
2 2 854905 1642 0 0 0 0 0 0 1896 6398 1536 38 36 26 0
3 2 854907 1592 0 0 0 0 0 0 1712 5466 1431 42 58 0 0
3 2 855232 609 0 0 0 0 0 0 1832 6864 1640 45 55 0 0
4 2 855232 537 0 0 0 0 0 0 1796 5170 1508 32 68 0 0
1 2 854907 884 0 0 0 0 0 0 1997 6052 1586 31 48 22 0
1 2 854907 856 0 0 0 0 0 0 2108 3944 1687 31 32 30 7
1 2 854907 851 0 0 0 0 0 0 2113 5408 1635 31 37 32 1
1 2 854907 852 0 0 0 0 0 0 2079 4084 1670 31 32 36 1
1 2 854907 852 0 0 0 0 0 0 2088 8426 1653 30 36 34 0
1 2 854842 902 0 0 0 0 0 0 1890 3600 1489 28 27 45 0



ftp> put "| dd if=/dev/zero bs=64k count=32000" /dev/null
2097152000 bytes sent in 106 seconds (1.933e+04 Kbytes/s)

Change2: (modify no -a to same settings as other machine)

>no -o udp_pmtu_discover=0
>no -o tcp_pmtu_discover=0
>no -o tcp_mssdflt=1448
>no -o nbc_pseg_limit=2097072

FTP TO TSM

root@asapdev1:/home/root >vmstat 5
kthr memory page faults cpu
----- ----------- ------------------------ ------------ -----------
r b avm fre re pi po fr sr cy in sy cs us sy id wa
2 2 854920 1376 0 2 2 284 1171 0 315 2418 630 7 14 74 5
2 2 854920 1372 0 0 0 0 0 0 2048 9011 1642 33 38 29 0
2 2 854920 1374 0 0 0 0 0 0 2050 4631 1630 36 35 28 1
1 2 854920 1360 0 0 0 0 0 0 2081 5758 1685 35 31 34 0
1 2 854920 1319 0 0 0 0 0 0 2065 4271 1947 34 33 33 0
1 2 854920 1320 0 0 0 0 0 0 2052 5658 1650 31 34 35 0
1 2 854920 1267 0 0 0 0 0 0 2027 4626 1682 32 33 34 0
1 2 854920 1266 0 0 0 0 0 0 2093 5536 1656 32 29 39 0
2 2 854920 1262 0 0 0 0 0 0 2078 4185 1674 31 34 35 0
2 2 854920 1251 0 0 0 0 0 0 2091 5587 1653 32 32 36 0
1 2 854920 1247 0 0 0 0 0 0 2061 4419 1643 32 39 29 0
2 2 854920 1246 0 0 0 0 0 0 2096 5459 1644 29 39 29 3
1 2 854920 1227 0 0 0 0 0 0 2091 4200 1703 32 31 36 1
1 2 854920 1227 0 0 0 0 0 0 2085 5478 1660 31 31 38 0
1 2 854920 1227 0 0 0 0 0 0 2077 3996 1653 31 31 38 0
1 2 854920 1223 0 0 0 0 0 0 2078 5547 1653 36 28 36 0
1 2 854920 1219 0 0 0 0 0 0 2067 3905 1663 32 31 38 0
1 2 854920 1219 0 0 0 0 0 0 2088 5604 1651 33 30 37 0
1 2 854930 1184 0 0 0 0 0 0 2041 4481 1709 36 34 30 0
1 2 854930 1168 0 0 0 0 0 0 2039 6204 1630 34 36 30 0
1 2 854865 1242 0 0 0 0 0 0 1692 4055 1341 28 32 40 0


ftp> put "| dd if=/dev/zero bs=64k count=32000" /dev/null
2097152000 bytes sent in 100.9 seconds (2.03e+04 Kbytes/s)



As you can see a slight improvement but I still think there is a problem. I have investigate the placement of the Gigabyte adapter on the machine and noticed that it is on the same bus as one of my two fibre channel cards?? I'm going to try and organise to move the card to another bus, as stated in the PCI adapter placement guide.

Any other ideas? I'll see how hte backups go tonight to see any real improvement.. thanks

 

[MoshiachNow]

Hi,

1.Please post the results of :

entstat -drt ent0 |grep –i error

2.Tried replacing the cable ?

3.Have checked the switch port settings ?

4.Tried a different GIGA switch ?

5.A good test would be to communicate between two machines with GIGA adapters over the crossed cable,then you eliminate the switch issue.

"Long live king Moshiach !"

http://www.7for70.com/

 

[octar]

note: The gigabyte adapter is ent2

>lsdev -Cc adapter | grep ent
ent0 Available 31-08 IBM 10/100 Mbps Ethernet PCI Adapter (23100020)
ent1 Available 40-58 IBM 10/100 Mbps Ethernet PCI Adapter (23100020)
ent2 Available 11-08 Gigabit Ethernet-SX PCI Adapter (14100401)

>entstat -drt ent2 | grep -i error
Transmit Errors: 0 Receive Errors: 0
No Carrier Sense: 0 CRC Errors: 0
Lost CTS Errors: 0 Alignment Errors: 0
Max Collision Errors: 0 No Resource Errors: 0
Late Collision Errors: 0 Receive Collision Errors: 0
Deferred: 0 Packet Too Short Errors: 0
SQE Test: 0 Packet Too Long Errors: 0
Timeout Errors: 0 Packets Discarded by Adapter: 0
No mbuf Errors: 0

The switch has been checked (but I'm not confident of the information I was given) detailed below:

Here is the configuration in the switch :

Port Name Status Vlan Duplex Speed Type
------- ------------------ ------------ -------- ------ ------- ----
Gi0/1 vers switch ANT_fr connected 1 Full 1000 1000BaseSX
Gi0/2 serveur ANTTSM01 connected 1 Full 1000 1000BaseSX
Gi0/3 serveur ANTDBS01 connected 1 Full 1000 1000BaseSX
Gi0/4 serveur Antaps01 connected 1 Full 1000 1000BaseSX
Gi0/5 serveur Antaps02 connected 1 Full 1000 1000BaseSX
Gi0/6 serveur Antaps03 connected 1 Full 1000 1000BaseSX
Gi0/7 serveur Asapdev1 connected 1 Full 1000 1000BaseSX
Gi0/8 serveur Antacc01 connected 1 Full 1000 1000BaseSX


Unfortunatly the backups lastnight after the changed only managed to increase by on hour :-(

I'm running some script rcp tests and will post as soon as I have the info..

 

[octar]

RCP TESTS:

The following script was run on each server, does this point to a network issue or a server issue??
(GBFILE is a 1GB file)

rcp $GBFILE anttsm01:/dev/null
for HOST in $(cat /home/root/.rhosts | grep _gb); do
echo "---- $(hostname) to ${HOST} --- on --- $(date)"
time rcp $GBFILE ${HOST}:/dev/null
done

IN SUMMARY

REAL anttsm01 asapdev1 antacc01 antaps01 antaps02 antaps03 antdbs01
anttsm01 18.29 20.48 14.09 14.69 13.33 16.76 15.49
asapdev1 27.71 31.94 26.81 27.1 27.47 28.2 104.71
antacc01 25.03 30.53 26.28 25 25.18 24.95 24.8
antaps01 20.27 21.43 19.91 21.13 20.28 20.11 20.19
antaps02 20.09 20.93 20.11 20.07 20.76 19.93 19.86
antdbs01 20.77 20.77 20.4 20.38 20.3 20.17 21.11

SYSTEM anttsm01 asapdev1 antacc01 antaps01 antaps02 antaps03 antdbs01
anttsm01 17.89 19.18 17.46 16.88 16.86 16.77 17.44
asapdev1 25.66 14.17 25.49 25.54 25.82 25.74 26.17
antacc01 23.57 23.03 20.89 23.3 23.4 23.61 23.27
antaps01 19.86 19.58 19.73 17.89 19.83 19.73 19.77
antaps02 19.76 19.62 19.71 19.74 17.03 19.62 19.65
antdbs01 20.28 19.98 20.09 20.1 19.84 19.95 17.49





root@anttsm01:/usr/local/scripts >dsh /usr/local/scripts/gb_test.ksh
anttsm01=> ---- anttsm01 to anttsm01_gb --- on --- Fri Oct 24 12:43:19 DFT 2003

real 0m18.29s
user 0m0.04s
sys 0m17.89s
anttsm01=> ---- anttsm01 to asapdev1_gb --- on --- Fri Oct 24 12:43:37 DFT 2003

real 0m20.48s
user 0m0.05s
sys 0m19.18s
anttsm01=> ---- anttsm01 to antacc01_gb --- on --- Fri Oct 24 12:43:58 DFT 2003

real 0m14.09s
user 0m0.10s
sys 0m17.46s
anttsm01=> ---- anttsm01 to antaps01_gb --- on --- Fri Oct 24 12:44:16 DFT 2003

real 0m14.69s
user 0m0.08s
sys 0m16.88s
anttsm01=> ---- anttsm01 to antaps02_gb --- on --- Fri Oct 24 12:44:33 DFT 2003

real 0m13.33s
user 0m0.00s
sys 0m16.86s
anttsm01=> ---- anttsm01 to antaps03_gb --- on --- Fri Oct 24 12:44:50 DFT 2003

real 0m16.76s
user 0m0.02s
sys 0m16.77s
anttsm01=> ---- anttsm01 to antdbs01_gb --- on --- Fri Oct 24 12:45:07 DFT 2003

real 0m15.49s
user 0m0.06s
sys 0m17.44s
asapdev1=> ---- asapdev1 to anttsm01_gb --- on --- Fri Oct 24 12:47:56 DFT 2003

real 0m27.71s
user 0m0.03s
sys 0m25.66s
asapdev1=> ---- asapdev1 to asapdev1_gb --- on --- Fri Oct 24 12:48:23 DFT 2003

real 0m31.94s
user 0m0.07s
sys 0m14.17s
asapdev1=> ---- asapdev1 to antacc01_gb --- on --- Fri Oct 24 12:48:55 DFT 2003

real 0m26.81s
user 0m0.06s
sys 0m25.49s
asapdev1=> ---- asapdev1 to antaps01_gb --- on --- Fri Oct 24 12:49:22 DFT 2003

real 0m27.10s
user 0m0.04s
sys 0m25.54s
asapdev1=> ---- asapdev1 to antaps02_gb --- on --- Fri Oct 24 12:49:49 DFT 2003

real 0m27.47s
user 0m0.07s
sys 0m25.82s
asapdev1=> ---- asapdev1 to antaps03_gb --- on --- Fri Oct 24 12:50:17 DFT 2003

real 0m28.20s
user 0m0.05s
sys 0m25.74s
asapdev1=> ---- asapdev1 to antdbs01_gb --- on --- Fri Oct 24 12:50:45 DFT 2003

real 1m44.71s
user 0m0.08s
sys 0m26.17s
antacc01=> ---- antacc01 to anttsm01_gb --- on --- Fri Oct 24 12:59:33 DFT 2003

real 0m25.03s
user 0m0.03s
sys 0m23.57s
antacc01=> ---- antacc01 to asapdev1_gb --- on --- Fri Oct 24 12:59:58 DFT 2003

real 0m30.53s
user 0m0.05s
sys 0m23.03s
antacc01=> ---- antacc01 to antacc01_gb --- on --- Fri Oct 24 13:00:29 DFT 2003

real 0m26.28s
user 0m0.06s
sys 0m20.89s
antacc01=> ---- antacc01 to antaps01_gb --- on --- Fri Oct 24 13:00:55 DFT 2003

real 0m25.00s
user 0m0.06s
sys 0m23.30s
antacc01=> ---- antacc01 to antaps02_gb --- on --- Fri Oct 24 13:01:20 DFT 2003

real 0m25.18s
user 0m0.04s
sys 0m23.40s
antacc01=> ---- antacc01 to antaps03_gb --- on --- Fri Oct 24 13:01:46 DFT 2003

real 0m24.95s
user 0m0.05s
sys 0m23.61s
antacc01=> ---- antacc01 to antdbs01_gb --- on --- Fri Oct 24 13:02:11 DFT 2003

real 0m24.80s
user 0m0.05s
sys 0m23.37s
antaps02=> ---- antaps02 to anttsm01_gb --- on --- Fri Oct 24 13:05:29 DFT 2003

real 0m20.09s
user 0m0.04s
sys 0m19.76s
antaps02=> ---- antaps02 to asapdev1_gb --- on --- Fri Oct 24 13:05:49 DFT 2003

real 0m20.93s
user 0m0.02s
sys 0m19.62s
antaps02=> ---- antaps02 to antacc01_gb --- on --- Fri Oct 24 13:06:10 DFT 2003

real 0m20.11s
user 0m0.07s
sys 0m19.71s
antaps02=> ---- antaps02 to antaps01_gb --- on --- Fri Oct 24 13:06:30 DFT 2003

real 0m20.07s
user 0m0.01s
sys 0m19.74s
antaps02=> ---- antaps02 to antaps02_gb --- on --- Fri Oct 24 13:06:50 DFT 2003

real 0m20.76s
user 0m0.08s
sys 0m17.03s
antaps02=> ---- antaps02 to antaps03_gb --- on --- Fri Oct 24 13:07:11 DFT 2003

real 0m19.93s
user 0m0.09s
sys 0m19.62s
antaps02=> ---- antaps02 to antdbs01_gb --- on --- Fri Oct 24 13:07:31 DFT 2003

real 0m19.86s
user 0m0.06s
sys 0m19.65s
antaps01=> ---- antaps01 to anttsm01_gb --- on --- Fri Oct 24 13:10:11 DFT 2003

real 0m20.27s
user 0m0.05s
sys 0m19.86s
antaps01=> ---- antaps01 to asapdev1_gb --- on --- Fri Oct 24 13:10:32 DFT 2003

real 0m21.43s
user 0m0.08s
sys 0m19.58s
antaps01=> ---- antaps01 to antacc01_gb --- on --- Fri Oct 24 13:10:53 DFT

real 0m19.91s
user 0m0.04s
sys 0m19.73s
antaps01=> ---- antaps01 to antaps01_gb --- on --- Fri Oct 24 13:11:13 DFT

real 0m21.13s
user 0m0.05s
sys 0m17.89s
antaps01=> ---- antaps01 to antaps02_gb --- on --- Fri Oct 24 13:11:34 DFT

real 0m20.28s
user 0m0.05s
sys 0m19.83s
antaps01=> ---- antaps01 to antaps03_gb --- on --- Fri Oct 24 13:11:54 DFT

real 0m20.11s
user 0m0.05s
sys 0m19.73s
antaps01=> ---- antaps01 to antdbs01_gb --- on --- Fri Oct 24 13:12:15 DFT

real 0m20.19s
user 0m0.03s
sys 0m19.77s
antdbs01=> ---- antdbs01 to anttsm01_gb --- on --- Fri Oct 24 13:16:01 DFT 2003

real 0m20.77s
user 0m0.03s
sys 0m20.28s
antdbs01=> ---- antdbs01 to asapdev1_gb --- on --- Fri Oct 24 13:16:21 DFT 2003

real 0m20.77s
user 0m0.05s
sys 0m19.98s
antdbs01=> ---- antdbs01 to antacc01_gb --- on --- Fri Oct 24 13:16:42 DFT 2003

real 0m20.40s
user 0m0.01s
sys 0m20.09s
antdbs01=> ---- antdbs01 to antaps01_gb --- on --- Fri Oct 24 13:17:03 DFT 2003

real 0m20.38s
user 0m0.03s
sys 0m20.10s
antdbs01=> ---- antdbs01 to antaps02_gb --- on --- Fri Oct 24 13:17:23 DFT 2003

real 0m20.30s
user 0m0.06s
sys 0m19.84s
antdbs01=> ---- antdbs01 to antaps03_gb --- on --- Fri Oct 24 13:17:43 DFT 2003

real 0m20.17s
user 0m0.03s
sys 0m19.95s
antdbs01=> ---- antdbs01 to antdbs01_gb --- on --- Fri Oct 24 13:18:04 DFT 2003

real 0m21.11s
user 0m0.04s
sys 0m17.49s


Also the no -a is equal on asapdev1 and antacc01

 

[slamcorp]

could you also please post a no -a
thx