Slow backups

[isom67]

I've recently started working in an environment that uses BEB 10 to perform backups. The target library is an 8 drive compaq library with DLT8000's.

There is one backup host which is attached across a SAN (EMC Connectrix) to a compaq MDR and then the library. This one host is responsible for backing up approx. 250 hosts mainly NT/2000 and Netware. Full backups on weekends are 2Tb with 500GB differentials each day.

The guys here have not been able to see any higher than an average throughput of 70Mb/m. We've even moved away from the VLDB format to a dedicated SQL server for DB.

Questions:
Opinions on above configuration?
Views on single backup host?
Anybody else with similar configuration?

Brian.

 

[davidmichel]

Way too little data for this type of problem.
Is that 70mb/m when 8 drives are all doing backups at the same time or when only one is backing up.
First verify the hardware configuration can handle it.
One Fibre connection going in to the MDR and two SCSI out is obiously not enough. There should be only two drivers per SCSI connection.
Use activity log to see the throughput for each session to see if it is target related.
Check a local backup to the remote. If local is slow start there.

 

[isom67]

Yes, 70Mb/m per job with 8 concurrent jobs active. We can see reasonable throughput when one job is active (400-500) but the throughput just drops off when additional jobs go active.

The fibre connection is 1Gb (single hba via san to MDR) and checking the stats I've never seen the utilization go above 50%. As for the MDR we have two drives attached to each SCSI bus (4 port output).

Can't figure out where the bottleneck is. Host doesn't seem bogged. SAN utilization doesn't appear high. As per previous post, we've offloaded the DB stuff to a SQL box.

 

[davidmichel]

1 job = 500mb/m
8 jobs = 560mb/m
So yes there is a bottleneck.
Two drives per SCSI port is good, but I'm not so sure about the one fibre port going in. 1gb provides a 100mb/s throughput which you would think is enough but I'm not so sure. As a first step make a second fibre connection or check with support for the manufacture of the mdr on what a real life throughput should be for eight jobs running at the same time through one fibre port.

 

[et0]

I have a similar problem with a W2K BEB 10.5 server fibre attached to a HP Surestore 4/40 with 4 DLT8000 drives. Single backup runs about 500 mb/min and adding jobs divide that throughput between the jobs. 2 scsi channels on the tape unit with 2 fibre connections from the tape unit to brocade, one fibre connection on server.

 

[davidmichel]

One more point to keep in mind.

If the target storage is seen as a local drive off the backup server the data path is SAN disk storage directly to SAN tape storage. If the targe volume is off another server the data path is via the network connection.

 

[isom67]

I find it interesting that 'et0' is having the same problem. Is 'et0' using an fibre to scsi MDR or fibre to scsi converters?

Over the last couple of days we've direct scsi attached 4 x DLT8000 (two per initiator) and we're seeing excellent throughput, avg 300-500 per job (4 jobs concurrent).

Our next step will be to direct attach our library and check it out.

Regarding David's suggestion, we're going to explore the second GBIC on the MDR but doing the math (and checking SAN port stats) we shouldn't be hitting the theoretical max for 8 x DLT8000.

Assuming 6mb native and 12 compressed (not that you ever achieve these)

8x12=96 which is close to saturating the single fibre.

Anyway, thanks for the suggestions, I'll check it out.

 

[davidmichel]

et0 is using the same thing. The MDR is just a Fibre Channel router, which is a device that has one or more Fibre ports on one side and multiple SCSI on the other. By any other name it is the same thing and the only way to connect a SCSI device to a Fibre Channel SAN.

With no more that two drives to SCSI port you are fine there. So that is why the question on the Fibre side of the MDR.

There is another option and that is to enable logging at the switch and reproduce the problem. The error log will record all the errors for the different ports and so can help identify were the problem is.

 

[bloxster]

500 MB/min is about the max throughput for a 100Mb/Full duplex connection. I suggest your bottleneck is a single LAN connection somewhere.