comparing values from different lines 2

[jkluesner]

Hello all,

I am a new (very new) user of awk and I am trying to write a if statement that will store a value in a variable, skip to the next line and store that same value (but different line) and then compare the two. If they are equal then loop again, if they are not equal then print the line. I will show you what the data looks like. I am looking at the hour/minute field. I want to be able to only print out the line when the first value on each minute changes. I want to skip the repeats of the minute. Thanks for the help.

05 09 2006 0627 26 59.9890-111 25.715 396
05 09 2006 0627 26 59.9790-111 25.696 397
05 09 2006 0627 26 59.9690-111 25.677 398
05 09 2006 0627 26 59.9580-111 25.657 399
05 09 2006 0628 26 59.9480-111 25.638 400
05 09 2006 0628 26 59.9370-111 25.619 401
05 09 2006 0628 26 59.9260-111 25.599 402
05 09 2006 0628 26 59.9150-111 25.580 403
05 09 2006 0628 26 59.9050-111 25.561 404
05 09 2006 0629 26 59.8950-111 25.541 405
05 09 2006 0629 26 59.8860-111 25.521 406
05 09 2006 0629 26 59.8770-111 25.500 407
05 09 2006 0629 26 59.8670-111 25.478 408
05 09 2006 0630 26 59.8580-111 25.458 409
05 09 2006 0630 26 59.8480-111 25.438 410
05 09 2006 0630 26 59.8370-111 25.418 411
05 09 2006 0630 26 59.8270-111 25.399 412
05 09 2006 0630 26 59.8150-111 25.380 413
05 09 2006 0631 26 59.8040-111 25.362 414


The hour/minute is the fourth field over looks like (0631)

This is what I have tried so far:

#!/usr/bin/awk -f

min = substr($0,14,1)
{next}
min2 = substr($0,14,1)
{
if (min != min2) print "$0"
else
next
}

 

[Annihilannic]

I often find myself doing using awk for the very same thing:

[tt]awk '$4!=last {print ; last=$4}' inputfile[/tt]

Annihilannic.

 

[jkluesner]

I tried adding this......it printed out the same values. I am using a shell script to use various awk lines. This is what it looks like:

#!/usr/bin/env bash


AWKPROGRAM=awk.txt
MINPROGRAM=min.txt
INPUTFILE=396_686.txt
OUTPUTFILE1=tmp.txt
OUTPUTFILE2=tmp2.txt
OUTPUTFILE3=tmp3.txt
FINISHFILE=done.txt

echo "filtering lines"

awk '/Received.*at/{print $0}' ${INPUTFILE} > ${OUTPUTFILE2}

#awk -f ${MINPROGRAM} ${OUTPUTFILE1} > ${OUTPUTFILE2}

#awk '((NR % 5) == 0) {print}' ${OUTPUTFILE2} > ${OUTPUTFILE3}

echo "reformating"

awk -f ${AWKPROGRAM} ${OUTPUTFILE2} > ${OUTPUTFILE3}

awk '$4!=last {print ; last=$4}' ${OUTPUTFILE3} > ${FINISHFILE}


echo "finish"

I added the line you suggested at the end. However it created a fine that looks the same as before with multiple minutes. Possibly I have screwed up?

Thanks!!

 

[Annihilannic]

When I run it against the data you posted I get this output:

[tt]05 09 2006 0627 26 59.9890-111 25.715 396
05 09 2006 0628 26 59.9480-111 25.638 400
05 09 2006 0629 26 59.8950-111 25.541 405
05 09 2006 0630 26 59.8580-111 25.458 409
05 09 2006 0631 26 59.8040-111 25.362 414[/tt]

Is that correct?

Annihilannic.

 

[jkluesner]

is last a certian command that you have defined? Is it built into the awk library? If I create a new file and do what you have done I only get the last line printed. I am thinking that it might be the "last" part.

Many thanks for your help with this......

 

[Annihilannic]

No, last is just a variable name. You could use "fred" instead of "last" and it would work the same.

What version of awk are you using and on what operating system?

Annihilannic.

 

[jkluesner]

I am using OS X on a Macbook pro and I am using the default awk that came with the install. Maybe thats the problem?

Thanks

 

[Annihilannic]

Unfortunately I don't have access to an OS X system to experiment with, hopefully someone else can help.

Annihilannic.

 

[jkluesner]

If it is a problem with my OS or version would somebody pls let me know.....Thanks!!

 

[Annihilannic]

I used a2p[/p] to convert my awk script to perl and it came up with this (some unnecessary bits removed)... maybe it will work for you?

[tt]#!/usr/bin/perl
while (<>) {
chomp; # strip record separator
@Fld = split(' ');
if ($Fld[3] ne $last) {
print $_ . "\n";
$last = $Fld[3];
}
}[/tt]

Annihilannic.

 

[jkluesner]

I do not know perl. I am assuming that I creat a .txt file with this code and then I am not sure how to call it in my shell script or how to set a input/output file. Thanks again for the help!!

 

[PHV]

I just wonder why you can't do ALL the processing with a SINGLE awk program transforming $INPUTFILE to $FINISHFILE ?

Hope This Helps, PH.
Want to get great answers to your Tek-Tips questions? Have a look at FAQ219-2884 or FAQ181-2886

 

[jkluesner]

I can....this is just how I built it step by step. Any suggestion on calling a perl script with a shell script and setting input/output files with perl.

Thanks

 

[feherke]

Hi

Use the switches, Luke... I mean, Annihilannic.

perl -pae '$_=""if$F[3]eq$last;$last=$F[3]' /input/file

Not nicer, just shorter.

Feherke.

http://rootshell.be/~feherke/

 

[PHV]

I also wonder why thinking that a perl script converted from a non working awk program should do the trick ...

 

[jkluesner]

Yeah......feherke that didn't work either. It just printed the last line in the file. I installed gawk thinking it is a problem with my version of gawk....no luck still the same results using gawk. All I need is a simple filter I can't seem to get it...ahhhh!!! Thanks for all the help guys. Is there another way to right what Annihiannic wrote using awk? Maybe a less efficient way?

Thanks!!

 

[PHV]

Perhaps last have special meaning for gawk ...
awk '$4!=l4 {print ; l4=$4}' $OUTPUTFILE3 > $FINISHFILE

Hope This Helps, PH.
Want to get great answers to your Tek-Tips questions? Have a look at FAQ219-2884 or FAQ181-2886

 

[jkluesner]

PHV,

I didn't work in either gawk or awk. I have tried both. Does the |4 have a special meaning in gawk.....it doesn't matter right?...just a variable name? Anyway if I change the variable name to whatever it still gives me the same output. The file it makes looks like the input file....no change.

Any suggestions?

Thanks!!!

 

[PHV]

What is the output produced by this ?
awk '{print $4}' $OUTPUTFILE3 > $FINISHFILE

Hope This Helps, PH.
Want to get great answers to your Tek-Tips questions? Have a look at FAQ219-2884 or FAQ181-2886

 

[jkluesner]

0627

Thats all it prints