Apply awk program "from file 00001 to file 06831" (getting awk to repeat task on 6000+ fil

[abhishgva]

Hello all!

Here's my question -> I have about 6000 files each with 9 columns, each columns containing about 2000+ rows. So, lots of data!
I have been able to successfully grab the 3 values that I need to extract from 1 file and to write those 3 values into an output.txt file.
I now need to be able to somehow get awk to accept multiple input files to do the same for the remaining 5999 files and write the 3 values it grabs to the SAME output.txt file. This would mean that by the end of it, I end up with 1 output.txt file with 3x6000 numbers in 3 neat columns.

I know that awk can accept multiple files in the following way: awk ’program’ input-file1 input-file2 etc. But clearly I can't write the 6000 names out manually.

Does anyone know of a smart way to tell it "Apply awk program from file Data00001.dat to Data06831.dat"

Thank you!

#!/usr/bin/gawk -f

BEGIN {
inSpill=0;
outSpill=0;
}

// {
if (($8>3000000)&&(inSpill==0)) {
inSpill=1;
start=$5
}
if (($8>10000000)&&(outSpill==0)) {
outSpill=1;
end=$5
}
lastRead=$5
}

END {
NumCoincParticles=(end-start);
percentage = NumCoincParticles/end;
print NumCoincParticles" "percentage" "lastRead" "start" "end;
}

 

[feherke]

Hi

Which version of [tt]gawk[/tt] ? With 4.0 and newer I would try to change [tt]BEGIN[/tt] and [tt]END[/tt] with [tt]BEGINFILE[/tt] and [tt]ENDFILE[/tt], then running the script as script.awk Data*.dat > output.txt

Feherke.
[link feherke.github.com/]

http://feherke.github.com/

[/url]

 

[abhishgva]

Hi Feherke,

I'm using GNU Awk 3.1.8 :S

 

[feherke]

Hi

Then try it like this:

[gray]#!/usr/bin/gawk -f[/gray]

[blue]FNR[/blue][teal]==[/teal][purple]1[/purple] [teal]{[/teal]
  whatever[teal]()[/teal]
[teal]}[/teal]

[fuchsia]//[/fuchsia] [teal]{[/teal]
  [b]if[/b] [teal](([/teal][navy]$8[/navy][teal]>[/teal][purple]3000000[/purple][teal])&&([/teal]inSpill[teal]==[/teal][purple]0[/purple][teal]))[/teal] [teal]{[/teal]
    inSpill[teal]=[/teal][purple]1[/purple][teal];[/teal]
    start[teal]=[/teal][navy]$5[/navy]
  [teal]}[/teal]

  [b]if[/b] [teal](([/teal][navy]$8[/navy][teal]>[/teal][purple]10000000[/purple][teal])&&([/teal]outSpill[teal]==[/teal][purple]0[/purple][teal]))[/teal] [teal]{[/teal]
    outSpill[teal]=[/teal][purple]1[/purple][teal];[/teal]
    end[teal]=[/teal][navy]$5[/navy]
  [teal]}[/teal]
  lastRead[teal]=[/teal][navy]$5[/navy]
[teal]}[/teal]

END [teal]{[/teal]
  whatever[teal]()[/teal]
[teal]}[/teal]

[COLOR=darkgoldenrod]function[/color] whatever[teal]()[/teal]
[teal]{[/teal]
  [b]if[/b] [teal]([/teal]inSpill[teal])[/teal] [teal]{[/teal]
    NumCoincParticles[teal]=([/teal]end[teal]-[/teal]start[teal]);[/teal]
    percentage [teal]=[/teal] NumCoincParticles[teal]/[/teal]end[teal];[/teal]
    [COLOR=chocolate]print[/color] NumCoincParticles[green][i]" "[/i][/green]percentage[green][i]" "[/i][/green]lastRead[green][i]" "[/i][/green]start[green][i]" "[/i][/green]end[teal];[/teal]
  [teal]}[/teal]

  inSpill[teal]=[/teal][purple]0[/purple][teal];[/teal]
  outSpill[teal]=[/teal][purple]0[/purple][teal];[/teal]
[teal]}[/teal]

[small][maroon]Warning[/maroon] The above code was not tested[/small]

Feherke.
[link feherke.github.com/]

http://feherke.github.com/

[/url]

 

[abhishgva]

Hi!

I'm not exactly sure how to implement this? I mean, I altered my .awk file as you suggested. But how do I now call it from within the Terminal so that it uses all 6000 files as its input files?
I used to call it as follows: awk -f 'Grab_values.awk' trigger_spill00001.dat > output.txt
(this would be for 1 input file only)

But now that we're trying to apply it to 6000+ files, what command exactly do i put in Terminal? Since the only format I know off for awk in Terminal is: awk -f 'awk program' inputfile1 > output.txt

Thanks!

 

[feherke]

Hi

Already made a suggestion on that in my earlier post :

running the script as script.awk Data*.dat > output.txt

Feherke.
[link feherke.github.com/]

http://feherke.github.com/

[/url]

 

[abhishgva]

Awesome worked perfectly!
Thanks a lot!