awk multiple file processing

Vital2 · Apr 25, 2012

Hi!

Could you please help to modify the following awk code (script) to shorten it. This code goes through different files and calculates the mean and standard deviation of the last value in the second column. Now it is done in ``stupid'' way, i.e., if I want to add one more data set I would need to change a parameter N everywhere and add manually corresponding file name. Furthermore, adding different data corresponding to a parameter w (omega) would require adding one more block of the code. Though, since all these blocks have a similar structure I wounder if it is possible to make it easier with awk, i.e., have an array of different w (omega) I would like to process, and a parameter N corresponding to a number of files I need to process.

Thank you for your help and time.

Code:

<html>
  <head>
    <title>Page</title>
  </head>
  <body>
awk	'{ Nu[f]=0; {Nu[f] = $2}} END { N=3; mu=0; for(i=1;i<=N;i++) {mu += Nu[i]} mu /= N;  sigma=0; for(i=1;i<=N;i++) {sigma += (Nu[i]-mu)^2} sigma = sqrt(sigma/N);
	print 2 " " mu " " sigma}' f=1 ./_TUS+/N=4096/w=2/OUTPUT_ED/pre f=2 ./_TUS+_1/N=4096/w=2/OUTPUT_ED/pre f=3 ./_TUS+_2/N=4096/w=2/OUTPUT_ED/pre > ./DATA/errors_tus+.data

awk	'{ Nu[f]=0; {Nu[f] = $2}} END { N=3; mu=0; for(i=1;i<=N;i++) {mu += Nu[i]} mu /= N;  sigma=0; for(i=1;i<=N;i++) {sigma += (Nu[i]-mu)^2} sigma = sqrt(sigma/N);
	print 3 " " mu " " sigma}' f=1 ./_TUS+/N=4096/w=3/OUTPUT_ED/pre f=2 ./_TUS+_1/N=4096/w=3/OUTPUT_ED/pre f=3 ./_TUS+_2/N=4096/w=3/OUTPUT_ED/pre >> ./DATA/errors_tus+.data

awk	'{ Nu[f]=0; {Nu[f] = $2}} END { N=3; mu=0; for(i=1;i<=N;i++) {mu += Nu[i]} mu /= N;  sigma=0; for(i=1;i<=N;i++) {sigma += (Nu[i]-mu)^2} sigma = sqrt(sigma/N);
	print 4 " " mu " " sigma}' f=1 ./_TUS+/N=4096/w=4/OUTPUT_ED/pre f=2 ./_TUS+_1/N=4096/w=4/OUTPUT_ED/pre f=3 ./_TUS+_2/N=4096/w=4/OUTPUT_ED/pre >> ./DATA/errors_tus+.data

awk	'{ Nu[f]=0; {Nu[f] = $2}} END { N=3; mu=0; for(i=1;i<=N;i++) {mu += Nu[i]} mu /= N;  sigma=0; for(i=1;i<=N;i++) {sigma += (Nu[i]-mu)^2} sigma = sqrt(sigma/N);
	print 5 " " mu " " sigma}' f=1 ./_TUS+/N=4096/w=5/OUTPUT_ED/pre f=2 ./_TUS+_1/N=4096/w=5/OUTPUT_ED/pre f=3 ./_TUS+_2/N=4096/w=5/OUTPUT_ED/pre >> ./DATA/errors_tus+.data

awk	'{ Nu[f]=0; {Nu[f] = $2}} END { N=3; mu=0; for(i=1;i<=N;i++) {mu += Nu[i]} mu /= N;  sigma=0; for(i=1;i<=N;i++) {sigma += (Nu[i]-mu)^2} sigma = sqrt(sigma/N);
	print 6 " " mu " " sigma}' f=1 ./_TUS+/N=4096/w=6/OUTPUT_ED/pre f=2 ./_TUS+_1/N=4096/w=6/OUTPUT_ED/pre f=3 ./_TUS+_2/N=4096/w=6/OUTPUT_ED/pre >> ./DATA/errors_tus+.data
  </body>
</html>

Annihilannic · May 2, 2012

How about wrapping it in a little shell code:

Code:

# empty the output file
> ./DATA/errors_tus+.data

for w in 2 3 4 5 6
do
	awk '
		{ Nu[f]=0; Nu[f] = $2 }
		END {
			N=3;
			mu=0;
			for(i=1;i<=N;i++) { mu += Nu[i] }
			mu /= N;
			sigma=0;
			for(i=1;i<=N;i++) {sigma += (Nu[i]-mu)^2}
			sigma = sqrt(sigma/N);
			print w " " mu " " sigma
		}
	' w=$w f=1 ./_TUS+/N=4096/w=$w/OUTPUT_ED/pre f=2 ./_TUS+_1/N=4096/w=$w/OUTPUT_ED/pre f=3 ./_TUS+_2/N=4096/w=$w/OUTPUT_ED/pre >> ./DATA/errors_tus+.data
done

Annihilannic
[small]tgmlify - code syntax highlighting for your tek-tips posts[/small]

Vital2 · May 3, 2012

Thank you very much, Annihilannic!
Actually I did the same (shell+awk) recently when I realized that this is the easiest way to do it.

Tek-Tips is the largest IT community on the Internet today!

Members share and learn making Tek-Tips Forums the best source of peer-reviewed technical information on the Internet!

awk multiple file processing

Vital2

Programmer

Annihilannic

MIS

Vital2

Programmer

Similar threads

Part and Inventory Search

Sponsor