bzip2 before tar

[ManagerJay]

I'm sure this is something simple I am missing, but I am not seeing it.

Basically, I need to write a script to create compressed archives, so users can save files to CDs, of approximately 700 MB when compressed.

My idea is, once I have a list of the files to backup, compress each file as it is added to the archive, and then check the total space. When the size is approximately 700 MB, create the next archive file.

Everything was going great, until I tried to compress a file and then send it to tar.

I used, bzip2 -c filename > tar -cvf archive.tar - , and several other variations with no luck. Any suggestions would be greatly appreciated.

Thanks,


Jay

 

[elgrandeperro]

First, it is | not >. > is strictly a redirect to a file.

Perhaps you mean to ADD it to the archive? tar -cvf will create a new archive everytime, so you will only have one file in the archive! That kind of makes tar useless.

You probably mean -a or -r to always append or replace.

There several options to "update", like replace or just append to the end. Read man tar. The danger with doing this is you are replacing images on a file...there is always a risk of doing the wrong thing, like one "c" will blast the tar. Also, you aren't recording the pathname so 2 files that are named the same and on different paths could be a problem if your script doesn't handle it.

gene

 

[kHz]

You want to tar one file that is already compressed?

 

[stefanwagner]

I understood "he needs independent archives" from the question.
Perhaps the "|" solved all questions?

seeking a job as java-programmer in Berlin:

http://home.arcor.de/hirnstrom/bewerbung

 

[elgrandeperro]

Come to think of it, this does not make sense. The
| only sends the data, you lose the file name.

The only option is to do the traditional:

tar rf archive.tar filename.bzip2 ....

or tar cf or tar uf

gene

 

[p5wizard]

Why not bzip2 the files into a temp dir until that dir is about 700MB in total space used, then tar the contents of that dir into a tarfile? Tar overhead should'nt be that high...

Go from here (untested):

#!/bin/ksh
filelist=/tmp/filelist #list of files with absolut paths
tardir=/var/tmp/tarfiles # where the tarfiles go
tarbase=tarfile
tarnum=1
tarfile=$tardir/$tarbase$tarnum.tar
tempdir=/var/tmp/tar$$
sidedir=/var/tmp/side$$
rm -rf $tempdir
rm -rf $sidedir
mkdir -p $tempdir
mkdir -p $sidedir
cd $tmpdir
while read filepath <${filelist}
do
 # get dir part of file
 filedir=$(dirname $filepath)
 # make sure dir tree exists under $tempdir for this file
 mkdir -p ${tempdir}/${filedir}
 # bzip2 the file
 bzip2 -c $filepath >${tempdir}/$filepath
 # count K used for tempdir
 Kused=$(du -ks $tempdir|awk '{print $1}')
 if [ $Kused -gt 700000 ]
 then
  # we're past 700MB, so move last file aside
  mkdir -p $sidedir/$filepath
  mv $tempdir/$filepath $sidedir/$filepath
  # tar the dir
  tar -cvf $tarfile .
  # create new tar name
  (( tarnum=tarnum+1 ))
  tarfile=$tardir/$tarbase$tarnum.tar
  # cleanup for next run
  rm -rf $tempdir
  mkdir -p $tempdir
  # pick up file that was set aside
  mv $sidedir/$filepath $tempdir/$filepath
  rm -rf $sidedir
  mkdir -p $sidedir
 fi
done
# create last tar file
tar -cvf $tarfile
# remove temp dirs
rm -rf $tempdir
rm -rf $sidedir

HTH,

p5wizard

 

[mrn]

What O/S are you using?

Mike

"A foolproof method for sculpting an elephant: first, get a huge block of marble, then you chip away everything that doesn't look like an elephant."

http://www.airport-parking-site.co.uk/

 

[ManagerJay]

Thanks.

What I have done is:

create a list of files to be backed up using find.

read from that list, creating a tarchive, until the tar file is approximately 850MB.

bzip that file.

continue until all files are read.

If anyone would like for me to post the code I am using, I would be happy to do so.

I am running this on FBSD 5.4.

Thanks for all the help, and for pointing out I would be losing path names if I bzip before tarring.

Thanks again,



Jay