Too Many Files to List with 'ls' 1

[tbohon]

I have a .ksh script on our AIX 5.x box which goes to a specific directory every hour and pulls the files to process and send to another server via ftp. However, when the number of files becomes large (tonight there were almost 6000 of them, other times there are zero or just a couple of dozen) my code

ls -1 > filelist.dat

fails with the 'list too long' error.

Is there another way to pull a directory listing when the directory has thousands of files in it? Also, at some point the script needs to 'mv' and 'cp' this same number of files so I need to know if there is another way to perform those actions on a large number of files as well.

I've tried 'xargs' and 'exec' without success - but admittedly I haven't ever used them before so I may not have understood what I was doing.

Any help appreciated.

Tnx.

Tom

"My mind is like a steel whatchamacallit ...

 

[Annihilannic]

I can't see how that command on its own would result in a 'list too long'. What is the code around it? Is it between $( ... ) or backquotes?

Annihilannic.

 

[p5wizard]

Yes, you'll need to show more of your script (especially the mv and cp part), because [tt]ls -1[/tt] will not be the culprit in this case. It might take longer and longer to perform the [tt]ls -1[/tt] but that's all.

E.g. if your script tries any one of these:

[tt]mv `cat filelist.dat`
mv $(cat filelist.dat)
mv $(<filelist.dat)[/tt]

you might run against a limit of the total argument length.
On AIX 53, you can extend the memory the shell uses for this argument list:[tt] smit chgsys #change ARG/ENV list size to 6 or 8 or ...[/tt] or [tt]chdev -l sys0 -a ncargs=6 #default value is 4[/tt] . But generally xargs can do the same trick with less memory because the increase in memory use by the shell is for all shell processes on the system... so:

[tt]xargs -I{} cp {} /path/to/copydir <filelist.dat
xargs -I{} mv {} /path/to/destdir <filelist.dat[/tt]


HTH,

p5wizard

 

[andy61]

I also can't see how ls -1 is the culprit, but if the output is not to tty then ls is equivalent to ls -1, so you could try substituting ls for ls -1.

 

[tbohon]

Here is the entire script that creates the file - it has

ls -ltr

in this latest attempt:

#!/usr/bin/ksh

cd /u/live/ftptemp
ls -ltr ../filesin/*.ps > files.list

And here is the error message I get when the number of files exceeds some value (not sure exactly what that is, unfortunately):

/u/live/ftptemp/fcopy.ksh[4]: /usr/bin/ls: 0403-027 The parameter list is too long.

Thanks again.

Tom

"My mind is like a steel whatchamacallit ...

 

[PHV]

cd /u/live/ftptemp
ls -ltr ../filesin | grep '\.ps$' > files.list

Hope This Helps, PH.
FAQ219-2884
FAQ181-2886

 

[tbohon]

I'll give that a shot - haven't tried this particular combination

Thanks!

Tom

"My mind is like a steel whatchamacallit ...

 

[tbohon]

PHV: Works as advertised!

One more question on this general topic if I may ...

This same script failed this morning with this error:

rm -f /interface/ftp/prod/hpf/files_out/hci/*
ksh: /bin/rm: 0403-027 The parameter list is too long.

I've been told by a 'casual' Unix user that I could replace the 'rm' statement above with

ls | xargs rm *

to overcome this problem.

However, since I'm not comfortable with my xargs knowledge and since I can't afford to lose a bunch of files, could I have your thoughts? In this case it was 483 files to be removed - could be many, many more depending on activity.

Thanks again.

Tom

"My mind is like a steel whatchamacallit ...

 

[Annihilannic]

Well, you'd want to be sure about being in the correct directory before using ls | xargs rm (note that the * shouldn't be there). This should do it:

cd /interface/ftp/prod/hpf/files_out/hci && ls | xargs rm -f

The && logic means the rm will not run if the cd fails.

Annihilannic.

 

[tbohon]

Annihilannic:

Thanks - I'll give that a shot. We just added a third facility to the mix so I'm getting lots and lots and LOTS of messages every hour now.

Appreciate it.

Tom

"My mind is like a steel whatchamacallit ...