Full backup takes too long for large volume of small files 1

[anhwright]

Hi All,

Netbackup is a new installation for us. We are running NBU 5.0 with Solaris 8 master/media server and fiber attached to STK L700E library with 4 LTO2 drives.

One of our 20GB unix filesystem has about 600,000 + files. It tooks 20hrs to do a full backup and about 1hr to do an increment backup (about 500 files).

We have added LOCKED_FILE_ACTION = SKIP in the client's bp.conf file, and this helps the increment tremedously (no more status 41 abend).
The CLIENT_READ_TIMEOUT and CLIENT_CONNECT_TIMEOUT are set at 600 secs.


Any suggestions ?

Thank you in advance.

 

[enterprisebu]

What are your kpbs on these incrementals? If they are low you search for the threads and the FAQ that talk about tweaking the communications buffers. I was able to take a 13 hour full down to 4.5 hours.

 

[comtec17]

If these are small file like 32k files then your backups will take a little longer then a large file. This is due to the block size on the tape. For example if your block size is 64k and your files are 32k, 48k, 2k, and so on, each small fill is written to each 64k block. The block does not get filled all the way so it has to fill it with raw space then move to the next file.

 

[rugby01]

Veritas uses a TAR streams to perform backups. it's very bad at streaming millions of small files. The scary part is - during a recovery - your going to get worse performance. One option I have - buy a Virtual Tape Drive (disk)that will increase the performance. The other option which make your life a little more work - compress the files into larger sections and backup the section. Recovery would be a two step process - but it will cut your window down.

 

[johnny99]

Using FlashBackup could also be an option (at least to get the backup part faster)

/johnny

 

[PGPhantom]

Most, if not all speed issues are related to the network. Do not use auto-negotiate, force 100/full on the NIC and the switch. Tweak the buffers as per the FAQ.

 

[anhwright]

Thank you all very much for your valuable inputs, I’ve read the 'Buffer Tuning FAQ' and made the suggested modifications and our overall backup window improve tremendously (thank you!). But as far as the 20GB UNIX filesystem that has about 600,000 + files, the performance is still BAD.

NetBackup Tech Support recommends doing a test to see if the performance bottlenecks point to disk or scsi sub-system. I'm going to give this a try. If I don't get anywhere with this, I will try rugbu01's suggestion.

Thanks,
Anh Wright

 

[PGPhantom]

For Unix systems set the NET_BUFFER_SZ and the SIZE_DATA_BUFFERS sizes to 262144 for both - See if that makes any difference.

 

[metcalfep]

We had the same problem on a Windows system
Setting network cards/buffers etc improved it but not enough

We multistreamed the directories that had thousand of files and used little space

One directory took up 1GB of disk space and had 60,000 files in it

Analyse the server and see what directories have the large amount of files inside it - it may be only a couple of directoies that are slowing down the backups

Create one job that backups up everything and exclude the appropiate directories on the client

Create another job that includes the directories you excluded previously

In the file list stream those directories
where the number of streams = the number of drives in the RAID set

NEW_STREAM
Directory_path-1
NEW_STREAM
Directory_path-2
etc.

 

[PGPhantom]

Twp things that a lot of people ovrlook ...

Disable virus software when backing up
Defragment your hard drive.

 

[jgordo03]

Netbackup no only lists out files, it also sorts the files and sometimes this can cause errors. To omit the sort of the directory and speed up the backup add this touch file /usr/openv/netbackup/dont_sort_dir on the master and the client.
I had one large filesystem with millions of small files over 1TB, it took 35 minutes to just do a long list.

 

[enterprisebu]

PGPhantom,

I recently also set the tcpsend and recv OS buffers up higher and this improved performance. Even though I told netbackup to throttle up the OS was still placing a limit on what NBU could do.

 

[creakyjoe]

i don't want to state an obvious (to me) or stupid reply but is there any way some housekeeping (ie major deletion of files) can be implemented. i have this kind of problem from time to time and some of those times i have to almost dictate stricter /or newer housekeeping to ease the situation. but as a previous poster mentioned - a disk based solution might be a partial solution. we aim to do something like this, but i've only bought into it on the proviso that we still have tapes/netbackup as the final destination, so we improve restore times/accessibility of data, but not negate our existing backup strategy. i hear that netbackup 5 or 5.1 is alleged to have disk staging but i'm wary of new features like that

Rich

 

[lenski]

The sugestion to break the backup into sections is good. Most bottle necks are at the disk. With lots of small files the problem is providing the tapes with enough work to keep spinning. Break the backup into for example 4 different jobs, set your tape drives to multiplex and experiment with multistreams. This way you will cut down the wait time at the tape drive. This type of work does need some experimenting to get just right, too many or too few streams can slow the backup, but only by trying different combinations will you find the perfect setup.

 

[letournp]

Hi there !,

I'm no expert, but my first idea would be to implement a raw image backup. One that reads the disk sequentially, (like Symantec Ghost does). There is an option you can add to NBU for that.

Go to their web site and check if you can test it.

Enjoy !