I need a duplicate line remover desktop software.
I have this text file -
After I run the software, I will get -
I have a software to do this alredy, but it's taking too much time and I want a multi-threaded software that can handle large files.
I have 100 txt files, each of them is 5 gb, I want to remove duplicate lines in all these files.
Software will be multi-threaded and will work very fast, and will use full potential of my ram and cpu processor, I have [url removed, login to view] ghz dual quad core and 8 gb ram, I want the software to use all the resources and clean up the file quickly.
My server is 64 bit and windows 2003.
I have alredy posted a project but reposting it as the guy couldn't make it.
I want it done in 1-3 hours, I don't want to waste time.