Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

  • Congratulations strongm on being selected by the Tek-Tips community for having the most helpful posts in the forums last week. Way to Go!

Apply awk program "from file 00001 to file 06831" (getting awk to repeat task on 6000+ fil

Status
Not open for further replies.

abhishgva

Programmer
Aug 10, 2012
4
CH

Hello all!

Here's my question -> I have about 6000 files each with 9 columns, each columns containing about 2000+ rows. So, lots of data!
I have been able to successfully grab the 3 values that I need to extract from 1 file and to write those 3 values into an output.txt file.
I now need to be able to somehow get awk to accept multiple input files to do the same for the remaining 5999 files and write the 3 values it grabs to the SAME output.txt file. This would mean that by the end of it, I end up with 1 output.txt file with 3x6000 numbers in 3 neat columns.

I know that awk can accept multiple files in the following way: awk ’program’ input-file1 input-file2 etc. But clearly I can't write the 6000 names out manually.

Does anyone know of a smart way to tell it "Apply awk program from file Data00001.dat to Data06831.dat"

Thank you! :)

#!/usr/bin/gawk -f

BEGIN {
inSpill=0;
outSpill=0;
}

// {
if (($8>3000000)&&(inSpill==0)) {
inSpill=1;
start=$5
}
if (($8>10000000)&&(outSpill==0)) {
outSpill=1;
end=$5
}
lastRead=$5
}

END {
NumCoincParticles=(end-start);
percentage = NumCoincParticles/end;
print NumCoincParticles" "percentage" "lastRead" "start" "end;
}
 
Hi

Which version of [tt]gawk[/tt] ? With 4.0 and newer I would try to change [tt]BEGIN[/tt] and [tt]END[/tt] with [tt]BEGINFILE[/tt] and [tt]ENDFILE[/tt], then running the script as script.awk Data*.dat > output.txt

Feherke.
[link feherke.github.com/][/url]
 
Hi

Then try it like this:
Code:
[gray]#!/usr/bin/gawk -f[/gray]

[blue]FNR[/blue][teal]==[/teal][purple]1[/purple] [teal]{[/teal]
  whatever[teal]()[/teal]
[teal]}[/teal]

[fuchsia]//[/fuchsia] [teal]{[/teal]
  [b]if[/b] [teal](([/teal][navy]$8[/navy][teal]>[/teal][purple]3000000[/purple][teal])&&([/teal]inSpill[teal]==[/teal][purple]0[/purple][teal]))[/teal] [teal]{[/teal]
    inSpill[teal]=[/teal][purple]1[/purple][teal];[/teal]
    start[teal]=[/teal][navy]$5[/navy]
  [teal]}[/teal]

  [b]if[/b] [teal](([/teal][navy]$8[/navy][teal]>[/teal][purple]10000000[/purple][teal])&&([/teal]outSpill[teal]==[/teal][purple]0[/purple][teal]))[/teal] [teal]{[/teal]
    outSpill[teal]=[/teal][purple]1[/purple][teal];[/teal]
    end[teal]=[/teal][navy]$5[/navy]
  [teal]}[/teal]
  lastRead[teal]=[/teal][navy]$5[/navy]
[teal]}[/teal]

END [teal]{[/teal]
  whatever[teal]()[/teal]
[teal]}[/teal]

[COLOR=darkgoldenrod]function[/color] whatever[teal]()[/teal]
[teal]{[/teal]
  [b]if[/b] [teal]([/teal]inSpill[teal])[/teal] [teal]{[/teal]
    NumCoincParticles[teal]=([/teal]end[teal]-[/teal]start[teal]);[/teal]
    percentage [teal]=[/teal] NumCoincParticles[teal]/[/teal]end[teal];[/teal]
    [COLOR=chocolate]print[/color] NumCoincParticles[green][i]" "[/i][/green]percentage[green][i]" "[/i][/green]lastRead[green][i]" "[/i][/green]start[green][i]" "[/i][/green]end[teal];[/teal]
  [teal]}[/teal]

  inSpill[teal]=[/teal][purple]0[/purple][teal];[/teal]
  outSpill[teal]=[/teal][purple]0[/purple][teal];[/teal]
[teal]}[/teal]
[small][maroon]Warning[/maroon] The above code was not tested[/small]

Feherke.
[link feherke.github.com/][/url]
 
Hi!

I'm not exactly sure how to implement this? I mean, I altered my .awk file as you suggested. But how do I now call it from within the Terminal so that it uses all 6000 files as its input files?
I used to call it as follows: awk -f 'Grab_values.awk' trigger_spill00001.dat > output.txt
(this would be for 1 input file only)

But now that we're trying to apply it to 6000+ files, what command exactly do i put in Terminal? Since the only format I know off for awk in Terminal is: awk -f 'awk program' inputfile1 > output.txt

Thanks!
 
Status
Not open for further replies.

Part and Inventory Search

Sponsor

Back
Top