delete 376K files from a list 3

[ksbrace]

I have a txt file with filenames going straight down. I need to delete them.

rm -f $(echo $(cat files_to_del.txt))

gives me arg list too long.

also tried

xargs rm -rf < files_to_del.txt

it doesn't do anything.

any help would be greatly appreciated.

 

[iggsterman]

if the names are one per line:

cat files_to_del.txt | while read fn; do rm -f ${f}; done

Before you do this, test what commands will run by adding echo, as in:

cat files_to_del.txt | while read fn; do echo rm -f ${f}; done

 

[SamBones]

You can use "[tt]xargs[/tt]" if you add a command line switch...

xargs [b]-n20[/b] rm -f < files_to_del.txt

That would make it do 20 at a time. That could be quicker with 376K files to delete. Deleting 376K files one at a time means "[tt]rm[/tt]" is called 376K times. Doing 20 at a time means "[tt]rm[/tt]" is only called 18,800 times. If "[tt]rm[/tt]" is an internal command, it probably won't matter, but if your shell is actually calling "[tt]/bin/rm[/tt]", then it could.

If the command line is still too long, you can lower the number (i.e. "[tt]-n10[/tt]" to do 10 at a time). Depending on how large each file name in the file is (including any path), you might be able to even bump it up to do more per "[tt]rm[/tt]".

 

[ksbrace]

rm -f
rm -f
rm -f
rm -f
rm -f
rm -f
rm -f
rm -f
rm -f
rm -f
rm -f
rm -f
rm -f
etc....

 

[SamBones]

Small typo, try this...

cat files_to_del.txt | while read fn; do echo rm -f ${[b]fn[/b]}; done

 

[ksbrace]

LOL, I'm sorry I didn't catch it either!

 

[iggsterman]

Good catch Sambones, Thank you. It's all my keyboard's fault.

 

[ksbrace]

so...I ran the echo first and got the correct output:
rm -rf PXPROD_7csoh8si_64_1
rm -rf PXPROD_7dsoh8si_101_1
rm -rf PXPROD_7hsoh8sj_101_1
rm -rf PXPROD_7lsoh8sk_101_1
rm -rf PXPROD_jks8868l_101_1
cat files_to_del.txt | while read fn; do echo rm -f ${fn}; done

but when I took the echo out....nothing was deleted. I tried doing a few manually to make sure it wasn't a priv issue and I was able to delete them manually.

Any ideas or things to check?

 

[iggsterman]

check that that you are in the directory where the files reside, or use full paths in that list.

 

[ksbrace]

Yeah, I checked that too. I am in the directory of the files. and the file looks like this:
PXPROD_bqsj2c03_77_1
PXPROD_bqsj2c03_78_1
PXPROD_bqsj2c03_79_1
PXPROD_bqsj2c03_80_1
PXPROD_bqsj2c03_81_1
PXPROD_bqsj2c03_82_1

how would I spit the output to a log file to try and look for any error(s)?

 

[iggsterman]

There is no output from the script? normally you would add to the end of the line:

>output.log 2>&1

I just noticed you are doing [pre]rm -rf[/pre] where 'r' is for recursive. Is it what you need? I added 'f' for forcing, no confirm deletions. So to avoid unintended consequences remove 'r'. To see if there are prompts, also remove the 'f' so the line would read:

cat files_to_del.txt | while read fn; do rm ${fn}; done >/tmp/output.log 2>&1

 

[SamBones]

Try this...

# Using head instead of cat to only try 20 while testing
head -20 files_to_del.txt | while read fn; do [b]ls ${fn}[/b]; done

If the file exists, it will just print its name. If the file isn't there where you expect it, it will say "[tt]ls: cannot access PXPROD_bqsj2c03_77_1: no such file or directory[/tt]". Or whatever each filename is. This is just to see if they are there.

Have you done a few spot checks to make sure the files in your "[tt]files_to_del.txt[/tt]" file are really there? Pick a couple filenames and grep the file for them. Your file could have become out of sync with what's actually in the directory. It may look correct with a quick glance, but it could be off enough to miss them all.

Do you have permission to delete the files. You will not only need permission on the file itself, but you will also need 'w' amd 'x' on the directory they are in (it needs to be updated that the file is gone). If it's your current default directory, then look at the perms for the directory '.' (that's one dot).

Also, where did the "[tt]files_to_del.txt[/tt]" originate? If it came from a Windows system, you probably have a carriage-return/line-feed pair ending each line. This always messes up things like this. Look for a utility called "[tt]dos2unix[/tt]", or something like that. That will "fix" a Windows file.

Also, the files could have invisible characters in their names. Try adding a '[tt]-b[/tt]' switch to an '[tt]ls[/tt]' to see any non-printable characters in the names.

Lastly, if you know you're in the right directory, and you've checked all those things, the server is haunted. Burn it. Then take the remains and bury in among the roots of a banyan tree under a full moon. It's best to just cut your losses and move on in cases like this.

 

[ksbrace]

dos2unix fixed it