a better sort

[Guest_imported]

I am sorting a comma delimited file on aix using the sort command, I need to sort on the 1-2 and 4th columns.

problem is that the file is 500+ mb and the sort fills /var/tmp (i wish you could specify a different fs) the admin added some more room for the filesystem but it still fails and I don't think added more is an option

does anyone know of a better sort program, I was thinkin perl or C, that could accomplish this, otherwise I might have to find some time to write my own

thanks in advance

 

[MikeLacey]

There's a Perl module that might be of help to you. I found it on

http://www.cpan.org

use File::Sort qw(sort_file);

sort_file({
I => [qw(file_1 file_2)],
o => 'file_new', k => '5.3,5.5rn', -t => '|'
});

sort_file('file1', 'file1.sorted');

Mike
________________________________________________________________

"Experience is the comb that Nature gives us, after we are bald."

Is that a haiku?
I never could get the hang
of writing those things.

 

[dchoulette]

You could also split the big files in smaller ones, sort the smaller files then merge (with sort) the already sorted files. It could be long but it should work (the merging should use much less memory than a true sort) if you have enough space on some disk to handle the duplication of your big file in smaller ones.

bigfile=./bigfile
splitfile=splitted

split -l 10000 -a 3 $bigfile $splitfile
rm -f $bigfile.sorted
touch $bigfile.sorted
for file in $splitfile*; do
    sort -n $file > $file.sorted
    sort -n -m $file.sorted $bigfile.sorted         > $bigfile.sorted.new
    rm $file $file.sorted
    mv $bigfile.sorted.new $bigfile.sorted
done

Of course you have to replace the -n flag of sort by your sort key on fields 1,2 and 4.

I tried this script on a 'small' file that can also be sorted directly with sort and the resulting files does not differ.

Another possible solution:
On HP-UX I see in the man page of sort:

-T dir      Use dir as the directory for temporary
            scratch files rather than the default
            directory, which is is one of the
            following, tried in order: the directory
            as specified in the TMPDIR environment
            variable; /var/tmp, and finally, /tmp.

Have you tried the -T flag or the TMPDIR variable ?

 

[Guest_imported]

sweet! the -T flag, I should have looked closer at the man page, that is exactly what I want, it works perfect thanks a ton dchoulette