Decimate a file

[ooids]

Hi All,

Anybody know of a command (or simple script) to decimate a file by a user specified amount?

e.g. I have a file that has 24 million lines and I only want to keep every 10th line.

Many Thanks,
James

 

[feherke]

Hi

sed -n '10~10p' /input/file > /output/file

[gray]# or if your sed has [b]-i[/b] option[/gray]

sed -i -n '10~10p' /input/file

Tested with GNU [tt]sed[/tt].

awk '!(NR%10)' /input/file > /output/file

Tested with [tt]gawk[/tt] and [tt]mawk[/tt].

Feherke.

http://rootshell.be/~feherke/

 

[columb]

A very simple script

IFS='
'
KOUNT=0
while read line
do
  [[ $KOUNT -eq 0 ]] && echo $line
  (( KOUNT += 1 ))
  [[ $KOUNT -eq 10 ]] && KOUNT=0
done < infile > outfile

but I'm sure someone will come up with an amazing awk or sed script!

Ceci n'est pas une signature
Columb Healy

 

[ooids]

Many Thanks ALL!

 

[feherke]

Hi

Columb, I really like your ability to solve the problems, but sometimes you use some tricks... No offense.

[s](( KOUNT += 1 ))
[[ $KOUNT -eq 10 ]] && KOUNT=0[/s]

((KOUNT=++KOUNT%10))

Tested with [tt]bash[/tt] and (pd)[tt]ksh[/tt].

Feherke.

http://rootshell.be/~feherke/

 

[columb]

feherke

No offense taken - I'm here to learn as much as anyone else.
Incidentally your construct works under RedHat bash but not under AIX 5.1 ksh




Ceci n'est pas une signature
Columb Healy

 

[PHV]

Amendment to Columb's script:
[[ $KOUNT -eq 0 ]] && echo [!]"[/!]$line[!]"[/!]

 

[feherke]

Hi

Than I also learned something : another difference between [tt]ksh[/tt] and [tt]pdksh[/tt]. Thank you for the information.

Feherke.

http://rootshell.be/~feherke/

 

[feherke]

Hi

Columb, may I ask what means "but not under AIX 5.1 ksh" ? Wrong result, no effect, error message ? Could you, please, tell me how AIX [tt]ksh[/tt] reacts to the below expression ?

((KOUNT=(KOUNT+1)%10))

Feherke.

http://rootshell.be/~feherke/

 

[columb]

I put together a test script

#!/bin/ksh

set -xv
kount=1
while [[ $kount -gt 0 ]]
do
  echo $kount
  (( kount=++kount%10 ))
done

and the output is

+ ((  kount=++kount%10  ))
+ [[ 1 -gt 0 ]]
+ echo 1
1
+ ((  kount=++kount%10  ))
+ [[ 1 -gt 0 ]]
+ echo 1
1
+ ((  kount=++kount%10  ))
+ [[ 1 -gt 0 ]]
+ echo 1
1
+ ((  kount=++kount%10  ))
+ [[ 1 -gt 0 ]]
+ echo 1

i.e. it looks like the preincrement isn't happening. Your revised version works a treat.

Ceci n'est pas une signature
Columb Healy

 

[feherke]

Hi

So that was the problem. Thank you.

Feherke.

http://rootshell.be/~feherke/

 

[PHV]

The ++ syntax in arithmetic expression became available with ksh-93

 

[feherke]

Hi

Thanks. Then [tt]pdksh[/tt] seems more closer to that.

But now I am really confused. As I understand, [tt]ksh[/tt]-93 was released in december 1993 and AIX 5.1 in may 2001. This means utility upgrades are so rare on Unixes ?

Feherke.

http://rootshell.be/~feherke/

 

[vgersh99]

Feherke,
it varies from company to company. For example, Sun is still shipping 'awk' circa '86.

vlad
+----------------------------+
| #include<disclaimer.h> |
+----------------------------+

 

[feherke]

Hi

Hmm... So is not a rare habit.

Now I took a look at a machine. The release file says Solaris 10, the [tt]uname[/tt] SunOS 5.10. It has really old [tt]ksh[/tt] and [tt]awk[/tt], I can not even find out their version. But beside those oldies, is a [tt]bash[/tt] 3.0... Did I mentioned that I am confused ?

Feherke.

http://rootshell.be/~feherke/

 

[Annihilannic]

Yeah, me too feherke, Sun seem to take an ultra-conservative approach to updating core OS utilities in case they break customer's scripts and annoy them. Personally I'm more annoyed by how out of date they are.

You can find more up-to-date ones elsewhere, for e.g. /usr/bin/nawk and /usr/xpg4/bin and /usr/xpg6/bin.

Annihilannic.

 

[segment]

Annihilannic:

No wonder one of my awk questions was way off. I ended up doing it in Perl (-pi -e) piped to sed. You know... For what its worth, I thanked some of you guys on a vulnerability this week I found on Asterisk about two months ago. I just made the disclosure public (

http://www.infiltrated.net/asteroid/),

but I asked a lot of questions that assisted me during my testing...



perl -e 'print $i=pack(c5,(40*2),sqrt(7600),(unpack(c,Q)-3+1+3+3-7),oct(104),10,oct(101));'