Smart questions
Smart answers
Smart people
INTELLIGENT WORK FORUMS
FOR COMPUTER PROFESSIONALS

Member Login
Remember Me
Forgot Password?
Join Us!

Come Join Us!

Are you a
Computer / IT professional?
Join Tek-Tips now!
  • Talk With Other Members
  • Be Notified Of Responses
    To Your Posts
  • Keyword Search
  • One-Click Access To Your
    Favorite Forums
  • Automated Signatures
    On Your Posts
  • Best Of All, It's Free!
Join Tek-Tips
*Tek-Tips's functionality depends on members receiving e-mail. By joining you are opting in to receive e-mail.

LINK TO THIS FORUM!

Add Stickiness To Your Site By Linking To This Professionally Managed Technical Forum.
Just copy and paste the
code below into your site.

Tek-Tips LinkedIn® Group

Partner With Us!

"Best Of Breed" Forums Add Stickiness To Your Site
Partner Button
(Download This Button Today!)

Feedback

"...Just to let you know...what a great site you have. I posted a pretty generic question yesterday and have had 8 responses already, anyway thanks again and keep up the good work..."
More...

Geography

Where in the world do Tek-Tips members come from?
Click Here To Find Out!

Tek-Tips Shirts!

Get Tek-Tips Swag
Get your
Tek-Tips Forums
SWAG here!
Home > Forums > Programmers > Languages > UNIX Scripting Forum

output only unique lines, no duplicatesHelpful Member!(3) 

thread822-1685164
Read More Threads Like This One
arunrr (Programmer)
6 Jun 12 23:33
Hello, I have the following input file.

Sri Lanka v Pakistan
England v West Indies
Sri Lanka v Pakistan
West Indies A v India A
England v West Indies
West Indies A v India A
England v West Indies
West Indies A v India A
Sri Lanka v Pakistan
England v West Indies
West Indies A v India A
Sri Lanka v Pakistan
Zimbabwe Twenty20 Triangular Series
West Indies A v India A
England v West Indies
Zimbabwe Twenty20 Triangular Series
Sri Lanka v Pakistan
England v West Indies
Zimbabwe Twenty20 Triangular Series

I need to only output unique lines as below.

Sri Lanka v Pakistan
England v West Indies
West Indies A v India A
Zimbabwe Twenty20 Triangular Series

Please note that I cant use "sort | uniq" as the order needs to be maintained.

Thanks
Arun
Helpful Member!  feherke (Programmer)
7 Jun 12 5:14
Hi

CODE

awk '!s[$0];{s[$0]=1}' /input/file
Tested with gawk and mawk.

Feherke.
http://feherke.github.com/

arunrr (Programmer)
7 Jun 12 7:02
Hi Feherke,

Thanks! That works great!!

Arun
mrn (MIS)
7 Jun 12 9:25
sort -u filename

Mike

"Whenever I dwell for any length of time on my own shortcomings, they gradually begin to seem mild, harmless, rather engaging little things, not at all like the staring defects in other people's characters."
Helpful Member!(2)  PHV (MIS)
8 Jun 12 4:21
Why not simply this ?

CODE -->

awk '!x[$0]++' /path/to/input

Hope This Helps, PH.
FAQ219-2884: How Do I Get Great Answers To my Tek-Tips Questions?
FAQ181-2886: How can I maximize my chances of getting an answer?

arunrr (Programmer)
8 Jun 12 7:34
mrn,

Thanks for your post. I cant sort as I need to maintain the order, meaning the first occurrence of a string must remain in its original position.

PH,

Your 'awk' statement works great. Appreciate it!

Thanks,
Arun
Read More Threads Like This One

Reply To This Thread

Posting in the Tek-Tips forums is a member-only feature.

Click Here to join Tek-Tips and talk with other members!

Back To Forum

Back To UNIX Scripting

Close Box

Join Tek-Tips® Today!

Join your peers on the Internet's largest technical computer professional community.
It's easy to join and it's free.

Here's Why Members Love Tek-Tips Forums:

Register now while it's still free!

Already a member? Close this window and log in.

Join Us             Close