Speeding up backup times?

[loade]

Hi,

Out of the blue I have been given the task of speeding up our unacceptably slow backups. I'm inexperienced using Bright Stor Manager r11.1 or anything related to backups.

Our current backup time with the powervault 132T is taking around 18 hours. About 16 of those 18 hours is backing up one server with about 410 GB on it. Here is the output of a log with that server being backed up.

Totals For................... \\*****
Total Session(s)............. 1
Total Directories............ 61,212
Total File(s)................ 770,368
Total Skip(s)................ 0
Total Size (Disk)............ 413,994.53 MB
Total Size (Media)........... 416,880.25 MB
Elapsed Time................. 16h 7m 9s
Average Throughput........... 431.01 MB/min
Total Error(s)/Warning(s).... 0/4

I was wondering if there is anything I could do to improve the backup time of this one server.


Thanks

Si

 

[loade]

Sorry, forgot some info that might help... As for the network, everything is local and on fiber. And the server being backed up is on a EMC Clarion.

Si

 

[vschumpy]

Hi, what drive do you have in the 132T? an average of 431Mb/Min isn't particularly slow, but of course it depends on what drive it is, what you are backing up, and what you expect and whether those expectations are based on manufacturer brochure figures, and whether those figures are for native or compressed throughput. This document will give you a good starting point for things to check which may help in the meantime:

http://supportconnect.ca.com/sc/redir.jsp?reqPage=search&searchID=TEC342813

 

[loade]

The drive is using LTO2, full uncompressed back ups. The files that are being backed up are users home directories.

As well, I was told that the 132T has two drives installed; does Bright Stor support more than one drive? And will I be able to use both of them to improve performance for that or other servers being backed up

I have seen averages of ~400MB/Min to ~630MB/Min without multiplexing. But these averages differ based off of network factors.

Thanks for the link, I'll look into that right away.


Si

 

[loade]

Arg, the link no longer works. Do you happen to have a copy of it?


Si

 

[blister911]

Try again, the link worked for me, although I did have to log into the CA website.

 

[vschumpy]

ARCserve supports more than 1 drive in the library, but you need to get out some cash and buy the tape library option if you want to be able to use both drives.

CA seem to want you to login to obtain the document - nothing I can do about that i'm afraid, they seem to have changed their policy again

I was a little confused when you said local and on fibre, but it seems as if you are having problems backing up a remote server across the network and it is just backing up this server that is the problem right? If so, you could perhaps look at this document:

http://supportconnect.ca.com/sc/redir.jsp?reqPage=search&searchID=TEC266855

It is a handy guide on how to troubleshoot network disconnection problems and other remote backup issues.

I'm assuming that you have the remote client agent patched up to the same level as the server - (you can check this in help about on both the host server and the client agent on the remote machine - they should both show the same version and build number - if they don't then let us know what you have in both places).

Multiplexing a single remote machine isn't really going to help you if the bottleneck is either network or disk bandwidth on this remote server - if you're lucky it might be something a simple as an old or improperly configured NIC driver or similar, if not, then please come back and give us a bit more detail on the remote machine's hardware -NIC, and setup.

 

[BackupFanatic]

An average speed of 413 Mb/min sounds slow to me for an LTO2 drive. You mentioned "full uncompressed back ups" - If you really have the drive set for no compression, you are only getting around half the tape capacity and potentially half the speed.

The other things I would look at are:

Defragging the SAN disk with the user files may help some. An automated defragger would be worth getting if a manual defrag helps the first time.

Update the firmware on your EMC SAN. They have regular updates and you may get both reliability and performance gains.

Check that all your NICs are set to the same speed & duplex. a setting of 'Auto' can be bad if one end decides to negotiate half duplex while the other goes full.

If you have spare switch ports, look at teaming extra NICs, particularly on the backup server. If you use multiplexing or disk staging, more bandwidth tothe backup server generally helps. If you really have dual tape drives, running concurrent jobs will save you time as long as you have enough network bandwidth.

Upgrade to Arcserver 11.5, sp1. It might not sound like a big change, but there are many fixes and improvements.

Look at disk staging. If your bottleneck is not the network or the source server, this can significantly decrease your apparent backup times.

Attach the backup server to the SAN and create snapshots of the slow fileserver onto the backup server's SAN partition. This can really improve speed if the bottleneck is now the tape drive.

Check your SCSI card. It is usually best if you have a dedicated card for the tape drive and definitely don't use a SCSI port on a card that already runs the server's disks.

If the problem comes back to the tape drive, look at upgrading to LTO 3 drive(s) They can be twice as fast (on paper) and you will have less tapes to change & manage.

 

[Grovie]

Further to BackupFanatics comments about NIC - the best settings are to set both the network switch port and NIC to Full duplex - these must be done together. (we've had this issue with 2 different versions of Brightstor/ASO)

We've also seen throughput issues when there are multiple backups running from the same RAID group in the SAN.

Have you checked the AV settings on the drives you're backing up and the backup server itself? In the case of eTrust, there are some settings which can significantly slow the backups in the realtime monitoring options.

It may also be worth going through the activity log in Job Status to check the job isn't stopping for hours at a time to wait for a file to unlock and to retry it - this can more than double backup times for the sake of one file, usually means BAOF is badly configured or simply not installed.

 

[gatom]

It makes me wonder....
First I have to tell you something regarding Brighstor.
All hardware must be compatible to do their work with BAB.You can check by yourself:
Open BAB manager/quickstart/Device-> klick on device and you will see the summary information.You can check SCSI ID and Devices with their firmware.
Now open this link to compare all(you have to look into certyfied tapelibary manufactors

http://supportconnectw.ca.com/public/storage/infodocs/basb9/basb9_cdl.asp

Check also firmware.
Now you should upgrade BAB to the latest build version:

http://supportconnect.ca.com/sc/kb/...&docid=377202&bypass=yes&fromscreen=kbresults

Follow this link:

http://supportconnectw.ca.com/public/storage/downloads/nt/v9/basb9-patchmenu.asp

By the way you need the Libary option.(only one drive in the libary is free.)Othersites, you can use n slots for free.
Please exclude BAB services from realtime AV scan.
Create a real system account for BAB.

http://supportconnect.ca.com/sc/kb/...&docid=266866&bypass=yes&fromscreen=kbresults

I'm sure this will help
GATOM

 

[vschumpy]

The PV-132T is on the CDL, and as the problem affects only one server it's highly unlikely to be a hardware or permissions issue affecting backup throughput in this case.

 

[kriskros]

Hi,
my experience with Brightstor 11.1 gives similar speeds to you. Using LTO2 library backing up scsi discs over gigabit network.
Backing up large single files e.g. databases will give you the fastest speeds e.g. 1 -1.5 gig /minute. Doing file servers gives about the same as what you're getting e.g. ~4-500mb/min. Someone told me that backup of many small files i.e. what you will find in home directories is fastest if backed up to disk. Haven't tried it but maybe a disk staged backup to disk then tape would improve things. Certainly doing a copy of the disk based brightstor backup file to tape will be quick.

 

[loade]

Thanks to everyone who posted any information regarding my problem. A lot of useful tips!

I ended up adding a second drive and splitting up the workload. I shaved the backup time to 9 hours total (7 hours for one drive, 9 for the other, running at the same time). I still think I can shave off some more time by adjusting the split and adding multiplexing. I'll let you guys know how it goes!


Si