How can I find duplicate records in a file using Unix. For instance, I have a file with employee information. How can I find records with duplicate employee numbers in the file.
If the records are sorted [sort -o file file],
then I suggest uniq:
uniq -d file
will give you a single instance of every duplicated (and triplicated, quadruplicated, etc) record. If you need to know how many times a record was duplicated:
uniq -dc file
Some uniq's support the -D option, spew out the second and subsequent duplicated records.
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.