Looking for ways to speed up backup

[kmcferrin]

Hello all, I am looking for suggestions about how to improve backup performance at my company. Following is a brief summary of our architecture.

We have approximately 25 servers, all running Windows 2000 Server SP4. We are using BackupExec for Windows Servers 9.1, with remote agents installed on all of the servers. I have two of the servers configured as backup servers, and the backup servers each have an Ultra320 SCSI adapter connected to an HP Ultrium 1 tape drive. All of my servers have gigabit ethernet NICs and are connected via a Cisco gigabit switch at 1 Gbps. The two backup servers each back up half of the servers.

Right now a full system backup takes 12-14 hours from start to finish, including verify. The full backups run from 8PM until completion, during the lowest period of system utilization. Most of the servers are multi-CPU with 3-4 GB of RAM and direct-attached RAID 5 arrays, so they should be more than beefy enough to run a backup. As it stands now, we can just barely fit the full backups on a pair of data cartridges (one for each tape drive). When reviewing my backup logs, I have been seeing a maximum backup speed of around 512 MB per minute, even when the server is backing up it's own local disks.

So my questions are this:

1. Is there anyway to actually get better throughput than what I'm seeing on my backups? Throughput of around 8 MBytes/sec wouldn't be suprising if I were backing up a remote server across a 100 Mbit connection, but it seems that's where I'm topping out in all circumstances (local Ultra320 arrays, 1 Gbit LAN connections, etc).

2. Right now the backup window is around 12 hours. I really can't let it go much longer than that because we're starting to hit production hours. If I can't get higher throughput than I'm already getting for backups, I may have to look into something like disk-based backups. But I'm reluctant to drop $30,000-$40,000 on a disk-based backup that will accommodate our current and future needs.

3. With the anticipated growth at my company, I suspect that we're going to start running into multiple tapes for the full backups pretty soon. I want to avoid having to change tapes in the middle of the night (as I do sleep). Going to an autoloader might be feasible if we can get better throughput (i.e., shorter backup windows), and that would be cheaper than buying a disk-based system.

What do you guys think?

 

[vm146971]

Before buying new hardware you should validate that you do not have a bottleneck somewhere that is preventing the tape drive from running at full speed.

You can expect 11-15MBps on LTO1 and faster w/ compression depending on the type of data and file characteristics.

There are many factors that could be slowing down the backup speed:
antivirus software: shut it off and run a test backup.
run a backup to disk and then run the same backup to tape and compare the results.
Copy files between the client and the backup server. Is there a significant difference between a copy and a tape backup?

Verify that the server NICS and the switch configurations are aligned correctly. Are there runts on the switch port?

Using latest firmware and drivers on the tape drives?
NIC drivers upto date?

The Capacity issue:
If all your backups currently fit on a single tape and you expect that soon you will need 2 tapes you could set a Cascaded Drive Pool. Basically to tape drives of the same capacity,(maybe also make and model)when one tape fills up it will automatically start writing to the next drive.

How much data do you backup in a week?

 

[steveot]

I agree with VM the fact that your getting around 512 mb min on remote backup is good with LTO. Yes a local backup with LOT should be a little faster.

As VM states there may be some bottle necks in the network.

Easy way to test is try some file copies of around 500 mb between the media server and remote servers.

Most common thing I have seen is drivers and NIC/PORT
Forcing them to 1g on the switch and NIC if possible usually helps out.

 

[kmcferrin]

I'll look into the NICs/ports/drivers and do the suggested testing. I'm just suprised that the throughput rate is so low, especially backing up local data.

Our current backup volume is a little over 300GB on the fulls once a week, about 40GB on the incrementals for the other 6 nights (compressed of course). But we've only been in business for a year or so, and we've got enough growth room in our systems that we could end up backing up several times that easily, so I'm looking for something that's fairly flexible for the next couple years.

 

[vm146971]

Are you running Veritas Backup Exec Compression? If so shut that off and let the tape drive handle the compression, that could be whats slowing you down.

 

[kmcferrin]

The compression option in Veritas is set to "Hardware (otherwise none)".

The drives are also using the Veritas drivers instead of the HP drivers. According to the Veritas install that would be the recommended config. Can anyone comment on the difference?

 

[jmcahren]

Dumb suggestion here:

Could you trim the backup load? We were backing up entire systems because we had time (and space), but as our data grew, we also began running long. We started backing up only specific data files, which saved a ton of time (and space).

For the record, we have a similar environment (gigabit, same backup device, raid5, etc., and I looked up last nights throughput rates, and for the five drives on one remote server (Essbase db), the following MB/Min rates were recorded: 812, 1300, 2138, 700, and 476. The drives with rates > 1000 store large data files.

30GB MSSQL bak file (remote): Throughput rate: 4503.4 MB/min

10GB Domino web server (remote): Throughput rate: 630.6 MB/min

5GB Domino web server (Local): Throughput rate: 227.2 MB/min

 

[kmcferrin]

Yeah, I would like to trim what we're backing up, but a lot of what we back up is governed by legal retention guidelines and we need the fulls for DR purposes.

The throughput rates that you're seeing seem impressive, but it makes sense that you get better rates on larger files. I'm still running some tests on my system, but I've noticed that in general you can copy a single 1 GB file across the network faster than 1000 1 MB files. To try to come up with something representative of our data I created a directory that has a mix of large and small files ranging in size from a few KB to 600 MB. The total size is 4.65 GB and a total of 4797 files. We'll see how it goes.

 

[kmcferrin]

Results so far:

4797 files, 4.65 GB of data total, file sizes ranging from a few KB to 650 MB.

Pull between two random servers: 476 MB/min
Push from media server to remote server: 656 MB/min
Backup to tape from remote server: 827 MB/min
Backup to disk from remote server: 602 MB/min

After seeing these results, I switched the NICs from auto to 1000 Mbit/Full Duplex, then ran some more tests.

Backup to tape from remote server: 926 MB/min
Backup to disk from remote serverL 588 MB/min

While some of these numbers are a little higher than I've been seeing before, none of them are dramatically higher
than what I had previously achieved. Since they were done during the afternoon I would expect that the performance was a tad lower than I would normally get in the middle of the night.

I guess my next step is to update the NIC drivers and see what I get. I'm using v6.34 and the most current are 7.80, so perhaps that will make a significant difference.

 

[cnozz]

I'm no Backup Exec pro but I use Veritas Netbackup. I had more servers to backup and a little larger operation, but these are the suggestions and questions I have for you. I only use 8 - 9840 tape drives in a StorageTek L-700 connected to 3 HP servers, 1 master and 2 media servers. The network I was backing up the clients on was 100Mb from the client to the switch and the from the switch 1Gb to the core backup switches.


Are you multiplexing to the Tape Drives?
How many tape drives are you using?
Why not do incremental backups for x-days per week then do full backups once or twice a week?

 

[kmcferrin]

No multiplexing. We only have two tape drives, each is connected to a separate server, each server is running a separate job.

We are doing incremental backups Tuesday through Sunday and full backups on Mondays. The root of the issue is that the full backups are getting to the point where A) they will be spanning tapes and B) they already take around 13 hours (including verify), and we're beginning to spill over into production hours.

The solution to issue A is to get autoloaders or a drive pool, but that leaves me with problem B which is much larger. I've done some digging, and I just can't seem to get throughput any higher than shown above no matter what I do (even backing up to disk!). I suppose that I could add a couple more tape drives and backup servers to the mix so that we're running 4 jobs simultaneously instead of 2. That would address both issues today, but this time next year I'll be hitting the same walls again. Surely there's a better way...

 

[cnozz]

Sounds to me that you have a network problem and you need a scalable option for a tape library. I ran into a wall in my datacenter and found that the switches for not keeping the configs correctly. I'd be interested in seeing a network diagram for what your doing. You are running all windows no Unix or anything else. Plus how much data are you backing up during your full backups.

Chris

 

[kmcferrin]

Kinda hard to diagram, but I can describe it.

All of the servers that are relative to this backup are Windows 2000 or 2003. There is one server running AIX and a pair of servers running RedHat Linux that are on the network, but are backed up separately.

All of the Windows servers (both doing backups and being backed up) are plugged into the same Cisco 3750 24TS+4 1Gbps switch. The switch is part of a cluster of 8 Cisco 3750 48+4 100 Mbps switches. However, the gigabit switch is configured as a single VLAN and all of the servers are on the same VLAN.

As far as data goes, each of the two backup servers is backing up roughly a dozen other servers. A full backup ends up taking up around 140-150 GB from each backup server for a total of around 280-300 GB.

 

[comtec17]

Make sure your Antivirus is turned off while you are backing up data. Most AV programs run both outbound and inbound scanning on files. This can be a very big issue when backing up network clients.

 

[ymeq123]

May be time to switch to LTO-2 drives? We just picked up an overland data 11 tape loader w/1 drive($3000 on ebay).

400GB per tape compressed connected to a Ultra160 SCSI.

I did a 168GB w/verify for 5 servers in 5 hours. I've cut the data down to 130 GB and no verify to get 3.5 hour full backup.

All servers have connection to 1Gb backbone and second NIC on 100Mb where the clients are. Veritas is using both NICs during backup the slow one for control and the fast for data.

I know it's not an answer to your question but it shows some real world times with the LTO-2 drives in a similar config.

(($30-$40K seems a bit high for a disk based solution. I came in way under half that with a two drive bay solution from IdealStor. (Of course the number of hard drives (Virtual Tapes) and drive bays you buy could drive the price way up. I was only looking at 5-10 drives. I don't think our future needs are as large as yours.))

 

[kmcferrin]

$30-$40k isn't too unrealistic when you're looking at multiple terabytes of storage space. I just spent a tad over $30k for a Clarion CX300 with approximately 2 terabytes of raw disk space (so you can see I'm going to need a better backup solution soon). We've been extremely impressed with the hardware that we've seen and purchased from EMC. You can't beat the features, support, and expandability but you do pay for it.

 

[cnozz]

You're correct EMC has some great storage solutions. But I don't think disk is gonna be a good choice for backups unless you do a lot of restores. I have a couple of suggestions for you on the full backups and expanding your backup capabilities. Now if you are looking for disk based storage for your servers, I'd look into something that fits
your prejected needs for the next 2 years that is scalable.
EMC is a good choice but I'd look into getting a refurbed system to help drive costs down.

1. Try breaking up the backup schedule for your full
backups.
IE: Do Half the full backups on Monday and the other half
on Tuesday. Just change the the incrementals to
reflect the changes you made to the Fulls.
2. Try looking into a robotic tape library for spanning
tapes.
IE: StorageTek L-20, L-40, L-80 they are scalable for
what it sounds like you'll need.

Chris

 

[ymeq123]

Yes KMC you a right. Every situation is different. EMC has some awesome stuff. You definitely get you you pay for. That solution would be overkill for my office but man would it be nice!!! I just wanted to give other options that I have looked into. You seem to be growing data quickly and you just just may need to bite the bullet and get something that powerful. I love playing with this stuff, it is so cool to see how much technology has advanced and made bakcup/recovery so nice compared to what it used to be!!!!!! I so wish I could have gone with a disk based solution my budget semms like it is on the negative numbers!!!! Ha!!!!!!

 

[cnozz]

To: Kmcferrin
You could always consider going to Netbackup to speed things up as well. Since BackupExec doesn't have the ability to multiplex to the tape drives it isn't as fast. But it is a trade off because multiplexing takes up more space on a tape. Hence my suggestion of a robotic tape library. You can buy for your needs and upgrade the library later with more drives and slots for tapes. It's just getting the growth projections correct for your needs.

To: Ymeq123
It's easy to say bite the bullet and get a very expensive solution, but you have to consider what the company has approved for your budget. Most times this is out of the administrators/engineers hands. A simple SATA disk array without all the goodies can make the solution cheaper but a little harder to manage.

Chris

 

[ymeq123]

CNOZZ,

Absolutly correct sir! That's what I was saying about my budget. That is why I went with the tape library solution instead of the IdealStor Disk backup solution I was looking at. The tape lib came in at half the price of the IdealStor and the tapes @ $40 a pop beat the snot out the price of extra disks and mounting trays!