backing up a LOT of little files

[trifo]

Hi!

I have a jfs2 filesystem, containing approx. 1.300.000 small files (~ 80GB sum). I have used tar to back them up, but as the sumber of files increased, now it takes about 15 hours to create the archive. savevg needs the same amount of time. Can you propose some solution to back it up effectively?

--Trifo

 

[unixfreak]

I'm guessing your filesystem is the problem... Not sure you can do much about it.

-----
Cheers,
Henrik Morsing

http://www.aixperts.co.uk/

Join us on irc://chat.freenode.net channel ##aix

 

[mrn]

Bit more info pls.

Backing up to Disk or Tape (if tape block size being used)
AIX Version
Server Type
Inline Log or seperate jfs2 log

Can you post the full command your using to backup.

output of "vmo -a" "ioo -a" "nfso -a"

to start with.....


Mike

"Whenever I dwell for any length of time on my own shortcomings, they gradually begin to seem mild, harmless, rather engaging little things, not at all like the staring defects in other people's characters."

 

[trifo]

mrn: OK, a do send now the rest of info:

The host is an LPAR of a pSeries 650 machine, AIX version is 5.2 TL 09CSP . The backup is created to disk, into an NFS-mounted filesystem. The jfs2 is using inline log.

The backup was created using GNU tar for years. Now this command takes 15 hours to run.

tar zpPcf /backup/vmail.tar.gz /vmail --exclude=/vmail/tmp

Now I tried savevg like this:

savevg -X -e -f /backup/mail_vg.backup mail_vg

#  vmo -a
       cpu_scale_memp = 8
data_stagger_interval = 161
                defps = 1
  force_relalias_lite = 0
            framesets = 2
            htabscale = n/a
    kernel_heap_psize = 4096
 large_page_heap_size = 0
         lgpg_regions = 0
            lgpg_size = 0
      low_ps_handling = 1
      lru_file_repage = 1
    lru_poll_interval = 0
            lrubucket = 131072
           maxclient% = 30
              maxfree = 700
              maxperm = 538739
             maxperm% = 55
               maxpin = 838861
              maxpin% = 80
      mbuf_heap_psize = 4096
      memory_affinity = 1
        memory_frames = 1048576
             mempools = 1
              minfree = 500
              minperm = 195905
             minperm% = 20
            nokilluid = 0
              npskill = 8192
              npswarn = 32768
     num_spec_dataseg = 0
            numpsblks = 1048576
         pagecoloring = n/a
      pinnable_frames = 630537
pta_balance_threshold = n/a
  relalias_percentage = 0
soft_min_lgpgs_vmpool = 0
     spec_dataseg_int = 512
     strict_maxclient = 1
       strict_maxperm = 1
             v_pinshm = 0
      vmm_fork_policy = 0

# ioo -a
                   hd_pbuf_cnt = 896
                    hd_pvs_opn = 3
         j2_atimeUpdateSymlink = 0
 j2_dynamicBufferPreallocation = 16
             j2_inodeCacheSize = 800
           j2_maxPageReadAhead = 128
             j2_maxRandomWrite = 0
       j2_maxUsableMaxTransfer = 512
          j2_metadataCacheSize = 800
           j2_minPageReadAhead = 2
      j2_nBufferPerPagerDevice = 512
j2_nPagesPerWriteBehindCluster = 32
             j2_nRandomCluster = 8
      j2_nonFatalCrashesSystem = 0
         j2_syncModifiedMapped = 1
       j2_syncdLogSyncInterval = 1
            jfs_clread_enabled = 0
             jfs_use_read_lock = 1
                    lvm_bufcnt = 18
                    maxpgahead = 32
                    maxrandwrt = 64
                 memory_frames = 1048576
                    minpgahead = 8
                      numclust = 16
                     numfsbufs = 196
                     pd_npages = 65536
            pgahd_scale_thresh = 0
            sync_release_ilock = 0

# nfso -a
                   portcheck = 0
                 udpchecksum = 1
              nfs_socketsize = 600000
          nfs_tcp_socketsize = 600000
           nfs_setattr_error = 0
        nfs_gather_threshold = 4096
         nfs_repeat_messages = 0
nfs_udp_duplicate_cache_size = 5000
nfs_tcp_duplicate_cache_size = 5000
    nfs_server_base_priority = 0
         nfs_dynamic_retrans = 1
            nfs_iopace_pages = 0
         nfs_max_connections = 0
             nfs_max_threads = 3891
      nfs_use_reserved_ports = 0
    nfs_device_specific_bufs = 1
           nfs_server_clread = 1
                 nfs_rfc1323 = 1
          nfs_max_write_size = 32768
           nfs_max_read_size = 32768
       nfs_allow_all_signals = 0
                 nfs_v2_pdts = 1
                 nfs_v3_pdts = 1
              nfs_v2_vm_bufs = 1000
              nfs_v3_vm_bufs = 1000
   nfs_securenfs_authtimeout = 0
   nfs_v3_server_readdirplus = 1
           lockd_debug_level = 0
           statd_debug_level = 0
           statd_max_threads = 50

--Trifo

 

[jprabaker]

Perhaps some kind of incremental backup system would help? Or possibly look to archive off older files if possible?

 

[ProbablyDown]

Maybe you can keep an open node that tars and compresses the files as they come in? I have used this when I had a f/s that didn't have enough space to tar and compress, but could it work in your situation?

mknod tar_pipe p
compress -f < tar_pipe > files.tar.Z &

Then have your files sent to the compressed file as needed:
tar -cvpf tar_pipe files/

Then when done:
rm tar_pipe

Its a band-aid, but may save time if it works..

 

[p5wizard]

Do you have enough local disk to try and make the tar file locally and then dump that tarfile to tape or remotecopy the tarfile to another system?

You may also want to check if it is possible to multi-thread this into different smaller tarfiles that you can create simultaneously? If you have suitable subdirs under /vmail ?


HTH,

p5wizard

 

[unixfreak]

Having multiple tars accessing the disk will probably just slow it down significantly. And technically, tar and savevg don't backup, they archive.

Only backing up changed file would help a lot, either a proper backup client like TSM or rsync it somewhere, or maybe just a find command to tar only the updated files.

-----
Cheers,
Henrik Morsing

http://www.aixperts.co.uk/

Join us on irc://chat.freenode.net channel ##aix

 

[p5wizard]

My guess is: most limiting factor is the NFS disk (what is the network between NFS client and server?)


HTH,

p5wizard

 

[trifo]

My guess is that backup would be faster it it used backbyinode, not backbyname. But I found myself unable to discover, how tho create an inode based backup using savevg.
I feel that most of the time is spent on searching the inode tables and opening/closing heaps of very small files. Maybe filemon would help to confirm this theory.

My other idea is to use mklvcopy/splitlvcopy to take a snapshot to run backup on, even using dd piped to a gzip process. This eliminates the file opens/closes and can take the advantages of sequential reading.

ProbablyDown: GNU tar can do this pipeline trick by itself. The -z option sends the stream through gzip.

--Trifo

 

[psi07004]

Have you considered using rsync to make the NFS copy, and then you can make the tar archive offline from that copy. If only a small number of the 1,300,000 files change, using rsync would take the copy time to almost nothing.

http://rsync.samba.org/

 

[trifo]

psi: Yes, I have considered this method, but it implies another problem: we are using TSM to backup/archive tasts, and it is not happy backing up a lot of little files. Anyhow, according to my observations, this is not the NFS which is the bottleneck in the process. The JFS itself won't read more than 3-400kB/s, even if archiving to dev/null.

--Trifo

 

[mrn]

Are you using Gigabit network adapters?

There is a known problem that is fixed in IBM APAR IY51569 (Packet trace shows pauses before resuming send), that could result in the poor performance.

IY51569 upgrades bos.net.tcp.client to 5.2.0.17 - bos.net.tcp.client 5.2.0.17.
IY51569 U496526 packet trace shows pauses before resuming send.

Also, if using a Gigabit adapter, there are two functions on the Gigabit card that should be turned off, to improve the performance.

lsattr -El entX
shows a field large_send and chcksum_offload. Set them to No.

ifconfig enX down detach
chdev -l entX -a large_send=no
chdev -l entX -a chcksum_offload=no
chdev -l enX -a state=up





Mike

"Whenever I dwell for any length of time on my own shortcomings, they gradually begin to seem mild, harmless, rather engaging little things, not at all like the staring defects in other people's characters."

 

[mrn]

Might also be worth upping the number of nfs daemons

ps -ef|grep nfsd

................/usr/sbin/nfsd 3891

stopsrc -s nfsd
chssys -s nfsd -a 4500
startsrc -s nfsd

Mike

"Whenever I dwell for any length of time on my own shortcomings, they gradually begin to seem mild, harmless, rather engaging little things, not at all like the staring defects in other people's characters."

 

[trifo]

mrn: the mentioned patch is already installed on the host. Anyways, can you tell me the reason to turn off checksum offloading and large send?

The host is connected to a gigabit switch, but the NFS server from which the target fs is mounted, uses only 100MBPS. I have checked the duplex mode, and it shows full duplex on both ends of the line.

There is no nfsd in the ps output, as this host is nfs client.

--Trifo

 

[foobar13]

Hi trifo,

You've mentioned considering backup by inode but said you didn't know how to get savevg to do this. I didn't see it mentioned that the backup command does this. You don't get to pick and choose which files you back up - you need to do the entire filesystem. This command appears to be similar to the dump command found on the other unices. Also, you can do incrementals by specifying the level, which would really speed things up for you.

You also mentioned that the destination host is sending these files to TSM. Doesn't TSM on AIX have an "image" option? I've never used it, but it might be worth investigation to see if you can get a backup by inode using TSM directly from the original server.

Foobar

 

[wpdavids]

I suggest you use "rsync" ... I use it for most file copies and to sync data. It makes a file list then starts your transfer. You can use it to sync or copy files on your local system or you can copy to other systems using NFS mounts (which I recommend). It also has the ability to use ssh or rsh as a transport method to other systems. The best part is if your "rsync" is interrupted for any reason you can start it again and it picks up where you left off. I've transfered millions of files with one rsync. It may run for about couple of days but eventually you get the copy done and can produce a log file of each file copied. It's been around for a while so try it.

 

[trifo]

wpdavis: well, I need to create one (or few) archive files to be sent to tape by TSM. Thus rsync is not suitable for my purposes. I use rsync to keep sync between multiple layers of web content with similar amount of files. For that task it rules.

--Trifo