Files taking a log time to back up...

[bTkalternate]

Hi all,

Can you tell me if there are particular file types that take longer to back up than others of the same size.
I'm mostly thinking of .pst (outlook archive) files.

Could you also tell me if it's normal for 3,500,000 totaling files to take such a long time to back up.
Here's a bit of the log:

Start time: 22:10:26 End time:03:25:23
Files: 3,504,511
Directories: 1,100
Byte count: 28,543,289,034 bytes
Rate: 86.00 MB/Min

Is it because there is so many small files that is causing the issue.
If so, is there a way to speed things up.

Many,many thanks,
Bren

 

[k3n85]

What type of interfaces are you using ? 100 mb or 1000 mb ? Or what type of hardware setup are you using? I do know that we use an 2nd backup network of only gigabit, with some systems or files taking a bit longer.

 

[bTkalternate]

Got a gigabit network using quality intel NICs and cisco switches. We backup when the LAN is not in use too so I think our infrastructure is OK.

Probably the pst files dragging things out.

 

[k3n85]

We use a Novell Groupwise mail system so I can't comment on the Outlook portion of the system (pst files) so might wait for feedback from someone else on that note.

 

[hmallett]

If you have lots of small files it will indeed take longer than a few large files.

http://www.BackupExecFAQ.com

 

[mattjurado]

Well, I think you know your bottlenecks here. You have gigabit interfaces, so that's fine (I'm assuming your ports on your switches are as well), and your backup drive.

Your LTO drives are the sh!t today.

Still I have always preferred to copy data to the computer that has the tape drive, THEN write to tape in a separate job.

A Linux box that uses Rsync (although there are comprable Windows tools) to get a full copy, then refreshes that copy every night with what has changed, is nice. That way, your nightly backup consists of syncronizing data on the wire, then write the whole thing to tape. Should be faster.

Matt

Please always take the time to backup important data and verify that backup, before making any changes suggested.

 

[Kahlor]

brendenmiller,
Our backup takes a long time to run too. We moved from Tapeware to Backup Exec 11d 2 months ago. Got new server, new tape drive.
I backup only the extremely important files during the week (500 GB, takes 17 hours) and then try to do a full backup on the weekend (1 TB, takes 40 hours)

Info on my daily backup
Files:549,675
Directories: 99,480
Byte count: 504,793,263,690 bytes
Rate: 1,378.83 MB/Min
Elapsed Time: 16:45:00

I'm still looking for a way to speed things up too.
1. From what I've seen, Backup Exec seems to like .pst files and large files. I have one area that has large files and I can get 2300 MB/min. I have another area that has lots of small files in lots of directories (on the same server) and I get about 170 MB/min.
2. When I look in Restore and browse the servers for files, I have noticed that some stuff is not getting backed up even though the job log acts like it is.
3. We use McAfee Viruscan and I have tried disabling the On Access scan during the backup. It helps a little bit, but not much.

 

[bTkalternate]

OK, I think I know what the problem is.
As we have all seen the logs here show small files take the time.

I think I read that the cause was from how BU Exec handles files.
It writes a 'start of file' then 'data' then 'end of file'. Each 'start' and 'end' will take a bit of time to do but the data will stream pretty quickly. Therefore having lots of 'start' 'end' will cause a slow down.
This theory makes sense.
I wonder if there is a way around this or if other back up software will handle things differently...

I've decided to offload the back up of the service with all the small files to a separate LTO3 drive that I had spare. Even though that hasn't sped anything up it has meant the job can run without taking up the time of the other job.

Good luck and I hope we can both find a real solution if one exists.

Brenden

 

[SteveRoadWarrior]

Quite a few of our clients are getting excellent results with backup to disk. Then make a duplicate copy of the backup set to tape.

The B2d location must be large, sure. But once your data is there, the tape gets to run at native speed with no "shoeshining". There's the advanced disk backup option too, but a duplicate copy operation seems to be a good way to deal with offsite storage. Especially if you get your full backups during off hours and only run differential jobs in between.

Try it out on a small section of your data to see what you think.

 

[Kahlor]

I'm starting to notice that my folder with lots of small files (963,600 files in 48,974 directories) will backup correctly if I turn Open File Manager off.

If I have OFM on, it seems to not back up those files. (Even though it looks like it did in the job log, I can go to Restore and browse the dates and see that those files are not listed - only the folders.)

That seems a little odd to me because no one is using those files at night, so I would think it would just copy them. But, at least now I know how I can backup that stuff. Now I can move on to my next problem in my backup job.