Eliminating duplicates from a long list ... 2

[dbeez]

Hi,

I am trying to eliminate duplicates from a comma-seperated list of email addresses that looks something like this (no newlines)

myemail@address.com,bossemail@address.com,
computerstaffemail@address.com,myemail@address.com

I have tried

sort -u -t , emailfile.txt

but it doesn't seem to work unfortunately ... what am I doing wrong ???

Thanks

BTW if anyone knows of a nice algorithm for awk that would do the job, that would be cool also

 

[Ygor]

Try...

awk 'BEGIN{ORS=RS=","}!a[$1]++' emailfile.txt

 

[feherke]

Hi

The [tt]sort[/tt]'s -t option is field separator, you need a record separator, which is not setable in [tt]sort[/tt]. To use [tt]sort[/tt], you must translate the commas in newlines, then back :

tr ',' '\n' < emailfile.txt | sort -u | tr '\n' ','

Feherke.

http://rootshell.be/~feherke/

 

[p5wizard]

Why not just convert the commas to spaces and let your mail gateway eliminate the duplicate addresses?

But I do like Ygor's awk solution though.


HTH,

p5wizard

 

[dbeez]

Thanks guys,

That's cool Ygor ... now if I could only understand it I'd be happier.

What is the best sed/awk/bash learning source for an aspiring programmer, do you think ???

 

[futurelet]

That's cool Ygor ... now if I could only understand it I'd be happier.

[tt]BEGIN{ORS=RS=","}[/tt]
This is executed before the file is read. Since the RecordSeparator is set to ",", Awk will read a field at a time instead of a line of a time.

[tt]!a[$1]++[/tt]
"!a[$1]" means if the array "a" has no value or a zero value for $1, then print the item just read. After this the "++" causes the value of "a[$1]" to be incremented by 1. The effect of all this is to print the item only the first time it is encountered.

For an introduction to Awk, see faq271-5564.

 

[Mag0007]

Good posts!

futurelet:

Thanks for the explanation!