removing duplicate lines ignoring empty lines and lines with #

[ogniemi]

hi,
how to achieve this without sorting input file?
only first occurrence of duplicates should stay.

following command found on a site does remove also empty lines and with #es which I don;t want to be removed.
perl -ni -e '$s{$_}++||print' file

so, lines like:

#####
(a whitespace)#(a whitespace)
(a whitespace)

should not be removed if such duplicates are found.

 

[PHV]

What about this ?
awk '!/^[ #]*$/ && ++s[$0]{next}1' file

Hope This Helps, PH.
FAQ219-2884
FAQ181-2886