optimise transferring data

[richw1234]

Hi there,

I need to move a large amount of small files on a weekly basis and would like to optimise this process. I currently use robocopy to copy the files across to a secondary server, both servers are HP DL380 G4 with Gig cards teamed. The servers are connected into a Cisco 2970 switch and Nic/ports are set to auto. I have configured the HP array configuration on the server which is pushing the data to 70% read and 30% write and vice versa for the receiving server. At the moment my throughput is only reaching 10mps, is there anything I can do to increase this? It's taking about 24 hours to push 350GB of data!

 

[juniper911]

A few suggestions...

could you transfer some files everynight?

could you zip/rar/tar them?

how about connected both servers directly using a crossover cables? should have better speeds or you could use quad teaming to really increase your throughput

 

[richw1234]

Hi there,

The answers to your suggestions are;

Yes, the data is being moved overnight.

Unfortunately I cannot zip them up as the files maybe needed to restore data, so this is out of the question.

I have tried connecting the servers with a crossover cable and this made a slight improvement but not enough.

I don't think adding quad teaming will better the performance as the current network utilisation is only peaking around 15% at times. And the throughput is registering at 10mps, this is not maximising the network as it is.

I think the problem is with disk I/O but I am not really sure on how to start troubleshooting or what to suggest.

 

[juniper911]

What is the RAID configuration of the target server? RAID 1 is slightly better than RAID 5, it has slightly better random and sequential write performance.

You can monitor the target servers hard disk performance using the perfmon tool and looking at avg disk queue lenth measurement.

Finally, what speeds are your hard disks? You can get 15k SCSI HDD's for the Proliants. There probably is other things you can do too but storage is not my strong point.

Finally, have you considered Fibre connectivity?

 

[itsp1965]

Things you may want to consider;
Instead of setting your switch ports/NICs to auto, consider having them hard coded them to run a 1.0GBps full

In terms of write performance, Juniper is correct in that copying to a RAID-1 drive is a lot faster than RAID5. If I am not mistaken setting up RAID 10 (Mirrored RAID 1 drive sets is even faster). 15K drives can't hurt either. Check that you are running the latest RAID/SCSI firmware,drivers (for your controller/SCSI drives). There could be an underlying issue that could be fixed by applying one of the above (check with the mfg)

Also what type of RAID controller are you using? is write cache enabled on the controller? And how much cache RAM do you have.

Hope this helps

 

[kmcferrin]

When dealing with servers, or any other devices that aren't likely to move from one switchport to another, ALWAYS statically set them for the fastest speed and duplex available. Because of the way the auto-negotiation works if there are negotiation issues it will drop to the lowest common denominator, which is usually 10Mbit/half duplex. It doesn't happen often, but it makes sense to avoid it if you can.

On top of that, keep in mind that you are moving a large number of small files. It takes longer to copy/move 100 1 MB files than it does to copy/move a single 100 MB file. If you're dealing with thousands of files, then you will see even more decreased performance.

Honestly, the best way around it is to have a higher performance disk subsystem. But before you spend the cash I would recommend doing some benchmarking on the drives that you have now to see if you're getting the kind of read/write performance during this operation that you would expect based on your benchmarks.

 

[richw1234]

Unforunately the hardware is already in place so I cannot do much about this at the moment. I can tell you that the primary server has SCSI disk in RAID 10 running at 15k rpm. The secondary is a sata array in RAID6ADG, these disk are running at 7.2k. The RAID controller on both servers is a HP6402 Cache ram is 192mb - the array accelerator is enabled but write back Cache is not.

I am a bit concerned about enabling write back cache due to the loss of data if there are any outages. But I am seriously considering enabling this, I am also considering cleaning of the RAID6 array and recreating a RAID10, I can change the port NIC speeds, this is not a problem. I will run through the firmware/driver versions to check for up dates.

I will change one of the above suggestions each day to see if the change has made a difference in performance.

 

[kmcferrin]

Rebuilding the RAID 6 array as RAID 10 may be helpful. RAID 5 and 6 are both relatively slow on writes when compared to non-partiy RAID. The fact that you're writing to a slower SATA array and that it's running a parity-based RAID could play a major factor. That and the potential for networking speed/parity mismatches are the two biggest potential bottlenecks that I see.

 

[richw1234]

I have recently changed the NICs/ports to full 1000 but this did not make a difference. I have now enabled the write cache on the remote server(sata disks) as this seems to be the bottle neck. Will keep you informed on its progress.

 

[kmcferrin]

Is the cache battery backed, or is the server on a UPS? That should mitigate some of the risk.

The other idea I had is the same as Juniper's. If your job zips the files up into a single .ZIP, preserving the directory structure, then unzips them on the receiving end, then you will get some improvement. The amount of data that you transfer will be reduced, which means it will go faster. Also, you'll be copying a single file rather than a bunch of small files, which should further improve performance.

Of course, that would be offset by the amount of time that it takes to zip and unzip on each end. But under certain circumstances that might be acceptable.

 

[technome]

If you only get 10 Mps, you most likely have a network issue. With a decent raid, proper network setup, certified wires, you should be pushing >110 with a gigbit NIC interface, no less more with properly teamed Nic interfaces.
If you network has SMB signing enabled, and internal security permits, disable it on wks and servers. If your wires are certified and you run at full duplex turn off flow control at machines and on switches; at least disable it on the machines, if the switches have it enabled..not necessary on both. If your wires are not certified, remember one bad wire or patch cord can seriously hamper an entire network. It is imperative the lines are tested with a device which can certify the wiring... cheap testers are useless. If you have managed switches get a report of packet/error losses on each port.


........................................
Chernobyl disaster..a must see pictorial

http://www.kiddofspeed.com/default.htm