Best way to remove duplicates in text file?

[mainit123]

I generate flat ASCII text files in a application where each row contains a full name e.g., John Q. Public. It is unavoidable that I will end up with duplicates of some names so naturally, I am looking to pare down the output file to only unique entries. While I know I could write code to look at each incoming read and compare it to what has already been read, I am asking if there is a function or method that is designed already to do just that?
Thanks.

 

[macleod1021]

If I was needing to do something like that, I'd probably set it up to load into a data table. Set the table up so that the name field has a unique requirement and then add error handling for when it attempts to add a duplicate. Either that or move from a flat file to a database where you can let the database engine handle the unique requirement.

 

[mainit123]

Thanks for the reply. I am sorry I neglected to mention that bringing in a database is not an option here. It is strictly a matter of doing string comparisons and I was looking to see if there was a known routine that was optimized for this task. I know I have my work cut out for myself when I am seeing that "John Q. Public" and "John Q. Public" are NOT equal when doing a regular string compare. It almost looks like a character by character inspection is needed versus just string comparisons. I know having a compound string (more than one word in the string being compared) only creates more work so that is why I wanted to at least ask and see if someone had done some previous legwork on this that was well-known.

 

[ThatRickGuy]

You don't need to have a database to work with a datatable.

-Rick

VB.Net Forum forum796 forum855 ASP.NET Forum
I believe in killer coding ninja monkeys.