What Unix Sort 'file-size' limitations are there?

[celconu]

I have several sort commands within a batch job stream. I am using stdin and stdout for most of the sort commands. The flat files that are being piped in and out are anywhere between 500MB and 30GB.

From what I noted in the man page, 'sort' is largefile compatible (2GB+).

I have changed the work file directory to a more suitable location and I have plenty of storage space. I have access to as much as 30GB of memory.

I am wondering if also, the piping of the process is causing harm (multiple sorts running at the same time). I really do not want to have to land this data before and after each sort.

If more information is needed, I will supply it.

Side note: I have used Opt-Tech's otsort in the past, but could use a manual if anyone has one.

Any help is appreciated.

Thanks.

 

[celconu]

EDIT:
The problem is that when I ran 100,000 records through this job stream... no problem. However, now that I have 2.2 million records, the sorts appear to be processing at about 1000 records per hour... not good.

Thanks again for your help.

 

[Annihilannic]

I'm not sure it will help, but have you tried using the -y kmem option to sort, specifying a generous amount of memory?

Annihilannic.

 

[celconu]

Thanks Annihilannic, I thought of that, and yet I hesitate to test it only because I used the TOP feature to watch the resource usage throughout the batch job and did not see the available ram drop below 20G.

I make the assumption that the sort command will take what it needs to perform the task.

Please correct me if I am wrong.

Thanks all.

 

[Annihilannic]

Actually, testing on Solaris 8 I discovered that -y is now obsolete, and you should use -S. What operating system are you using?

sort will take what it needs to perform the task, but if you give it more than necessary it may happily use it. No harm in trying!

Annihilannic.