Removing duplicate words in a file

[ianicr]

I have a file that is:
"MR","I","SMITH","1","NEW STREET","LONDON","LONDON"

I wish to remove the first london but leave the second. Is there an easy way to do this?

 

[SteveR77]

Question -

Given the format of the data being

"MR","I","SMITH","1","NEW STREET","LONDON","LONDON"


You would want to keep all the data in the case of a record

like this one -

"MR","R","JONES","1","NEW STREET","GREENWICH","LONDON"

Correct?

 

[ianicr]

Yep. That's just what I need. Not too bothered what goes into the field as long as its not the same as the field after.

 

[Ygor]

Perhaps use sed....

sed 's/\(,"[^"]*"\)\1/\1/g' file1 > file2

 

[aigles]

If you want to remove field 6 ("," separator) if it is the same as field 7 ...

with sed :

sed -e /s/^\(\([^,]*,\)\{5\}\)\([^,]*\),\3$/\1,\3/' input >output

with awk :

awk 'BEGIN {FS="," ; OFS=","} $6==$7 {$6=""; print $0}' input >output

Jean Pierre.

 

[SteveR77]

If you need to preserve the field place holder but not the value that it contains you could do the following -

sed 's/\(,"[^"]*"\)\1/,""\1/g' file1>file2

 

[PHV]

Try this:

awk -F, '
$6==$7{$6="\"\""}
{print}
' </path/to/inputfile

Hope This Help
PH.