Duplicate lines

[sharvey99]

Can anyone help

I have a file that has over 67000 lines and I need to find
any duplicates. The file format is

ipaddress hostname serial node

eg

10.1.23.1 A19.253c 032403EV02 H001a

It does'nt matter if it's takes time to run, I have time to
wait. I just need to know.

 

[bigscouse]

Try this: -

for i in `cat filename`
do
if test `grep $i filename | wc -l` -gt 1
then
echo "The line "$i" has `grep $i filename | wc -l` entries"
fi

 

[flogrr]

Looks like Ged forgot to add the "done" to the end of
his script. It works then but processes each field as
if it were a filename resulting in a ton of output.

I piped the output into sort -u like this:

script filename | sort -u > outfile

This gave just the unique report lines from the echo
command, but the output still requires you to sort it
out visually.

However, try this:

uniq -d filename > outfile

This will output a single copy of any repeated lines.

Hope this helps.

 

[bigscouse]

ooops, well spotted flogrr.

Sharvey, don't forget to "sort" prior to using "uniq" as it only looks at adjacent lines.

 

[gregor weertman]

while read x
do
if [ `grep -c $x b ` -eq 0 ] ;then
echo $x
fi

done < orgfile > destfile

Regards [sig][/sig]

 

[gregor weertman]

Oops

while read x
do
if [ `grep -c $x outputfile ` -eq 0 ] ;then
echo $x
fi

done < inputfile > outputfile [sig][/sig]

 

[gregor weertman]

the script above will only make an uniq file.

while read x
do
if [ `grep -c $x inputfile ` -gt 1 ] ;then
echo $x
fi
done < inputfile > duplicates [sig][/sig]