What command used to remove duplicate lines 1

[wchuacs]

Hi,

Anyone knows what is the command to remove duplicate lines in Unix ?

For example, I grep a file and put it into a new file and it contains some duplicate lines, I totally no ideas how to remove those dup lines to make my file more readable.

thanks for sharing.

regards,
CS thanks & regards,
CS

 

[KenCunningham]

If you have a file <filename> consisting of, say,

1
2
2
3
4

and run the command:

grep 2 <filename> | uniq > <newfilename>

that should do the trick. Hope this helps.

 

[KenCunningham]

Alternatively, use the following script to first of all sort the lines in the file, then test for duplicates and output only one line to the new file:

rm newfile
set oldline=&quot;&quot;
cat test | sort | while read line
do
echo $line
echo $oldline
if [ &quot;$line&quot; = &quot;$oldline&quot; ]
then
echo &quot;Line the same as previous one&quot;
oldline=&quot;$line&quot;
else
echo $line >> newfile
oldline=&quot;$line&quot;
fi
done

This of course assumes that you want the file sorted when output. Good luck.

 

[KenCunningham]

Sorry - just realised the 'set' in the second line of the above script is entirely superfluous, please ignore it! 'test' should, of course be replaced by your filename.

 

[wchuacs]

Hi KenCunningham,

Your scripts are excellent helpful. Thanks a lot.

CS thanks & regards,
CS

 

[prv3116]

An even simpler way, if the lines are indeed exact duplicates, use the sort command with the -u flag. Also use the -o flag to actually put it into a new filename or even back into the original filename. If you don't use the -o it will just output to your screen.

sort -u -o originalfile newfile

or

sort -u -o originalfile originalfile

 

[KenCunningham]

Nice one. Elegance has never been my strong point!!