how to back up milions of small files to tape for Linux

[talam]

We need to back up milions of small files to a RedHat EL5 server, which will replace an existing AS400 system. The RedHat EL5 server has a local LTO tape drive connected for backup. Currently it took 60-70 hours to backup 700GB files to tape (or disks). Disk IO and tape IO are not concern. The problem is related to millions of small files. Is this known problem for backup of millions small files? What is the best or workable solution... Thanks!

 

[Kozusnik]

I'd write a program to ftp the files over. Either a php/perl/shell script, or a CL program. Are the files on the IFS, or in a library?

If they're on the IFS, you could mount the share on the Linux server and just copy.

Mark

 

[elgrandeperro]

Do you write the tape and restore it?

Typically, either one would use a SAN mirror (if available) or rsync.

 

[Kozusnik]

I don't think rsync exists for OS/400. Maybe if you had a separate LPAR and installed some *nix OS, that would definitely be the way to go.

Mark

 

[Annihilannic]

The way I understand the question, you already have these files on the new RHEL5 server, right?

What commands/tools are you using to perform the backup?

Have you tried backup to another disk to determine whether the bottleneck is reading the files or writing to tape?

Annihilannic.

 

[MikeHalloran]

I think perhaps the question has to do with, e.g., whether the directory for, say, a million small files would fit on a tape, even if the files would fit... or if there's some other tape- specific glitch awaiting someone who entrusts important data to tape.

I honestly don't know. I gave up on tape years ago. Floppies are more reliable.

 

[elgrandeperro]

First, he said AS/400, not OS/400. I believe Linux (some form) runs on AS/400.

I have not seen (in the last few years) any format problem with millions of files (I have written such backups and restored).

Are you trying to read the file through the filesystem itself (like tar) or through the raw device (dump/restore)?

Generally, what happens is if the files are in a huge directory it spends lots of time stat-ing through the filesystem inodes as opposed to reading actual data. This does not keep the tape buffer full, which is suboptimal for a tape drive.

 

[Kozusnik]

You're right there are several versions of Linux which run on the iSeries. I assumed he was backing up OS/400 files to tape (IFS or Lib), then restoring to a Linux (Intel) server. If he isn't answering these questions, I guess we'll never know.

Mark