I need a duplicate line remover desktop software.
I have this text file -
After I run the software, I will get -
I have a software to do this alredy (2 of them), but 1 can only take 1 file at a time but work very fast, the other can take many files at a time but work very slow, and I want a software that take many files at a time and work very fast.
I have 2300 txt files, each is 100 mb, I want to remove duplicate lines in all these files.
I will open the software, click browse, select folders (I have 83 folders, each folder have 27 txt files).I will click delete duplicate, and software will delete duplicate on all [url removed, login to view] will treat each file seperately when looking for [url removed, login to view] will be multi-threaded, and it will dedicate 1 thread per file, if total there is 2300 files, software will run with 2300 threads, and software will work fast also.
Software will use full potential of my windows server resources - ram and cpu processor, I have [url removed, login to view] ghz dual quad core and 8 gb ram, I want the software to use all the resources and clean up the file quickly.
My server is 64 bit and windows 2003.
I have alredy posted a project but reposting it as the guy couldn't make it.
I want it done in 1-2 hours, I don't want to waste time.
I hope I have describe exactly what I wanted, I will give u 2-3 files (zip it so it becomes small) so u can test.
I have low budget, will choose the one who can charge low.