Develop a duplicate content checker that will work in the cloud

Hello, We need a tool to analyse large batches of files (100K to 200K files). Each file will be an article. The tool must detect the following "sliding" n-grams: 3-grams, 4-grams, 5-grams and 6-grams. Here is an example sentence : « de demander différents devis sur le site de Buuyers » This sentence is in French. It’s formed of height 3-grams shingles: 1. de demander différents 2. devis sur le 3. site de Buuyers 4. demander différents devis 5. sur le site 6. site de Buuyers 7. différents devis sur 8. le site de The tool must be able to detect all these different n-grams by comparing each article to all the other articles in the batch of 100K to 200K articles we'll upload. Then, it will calculate the similarity ratio, based on the Jaccard's index. So, let's imagine we have 105 3-grams in common between 2 articles making 1026 words together, we should calculate their similarity ratio this way : 105 ---------- = 0.11 = 11% of similarity between these 2 articles. 1026-105 We must be able to add a list of stop-words in French and in English. These stop-words would then be excluded from the analysis. If for example "the", "this", "is" are listed as stop-words, the sentence "this is the world champion" would count as only 2 words: "world" and "champion". Once the comparison is done between all the articles, the tool must be able to extract the most different ones. We'll ask it to extract all the articles with a maximum similarity ratio of 15%, for example. We must get a table showing how many articles we can extract, based on different maximum similarity ratio values, by increments of 1%. So, for example: we can extract 0 articles having a maximum similarity ratio of 0%, 1%, 2%, 3%, 4%, 5% and 6%, 17 articles at 7%, 196 articles at 8%, 1345 articles at 9%, 7635 articles at 10%, etc. A table will show how many articles we can extract, based on their maximum similarity ratio. A detailed view will enable to see the most similar articles, with their common n-grams highlighted. Example of comparative view: [login to view URL] So, user would ask the tool to see articles having a similarity of 16%, for example, and the tool would then show random articles having this similarity ratio. It will enable user to visualise them. We must get a report showing the most redundant shingles by n-grams among all the articles: 3-grams, 4-grams, 5-grams and 6-grams. With a percentage indicating in how many articles they were found. The tool must be able to automatically delete the articles having a similarity ratio superior to a value that user might be able to set and adjust for each batch of file. We must be able to load the batch of articles in zip archives. And to download the extracted articles in this format too. Let's say we upload 200K articles in 1 zip file, and then, after the analysis, we want to keep the 8'736 articles with a maximum similarity ratio of 11%, we'd then get them in 1 zip file too. The tool must show a counter to indicate how long ago it started and estimate the remaining time until completion. A mail must be sent to several email adresses when the task is completed, or if there's any error/problem during the process. Due to the nature of the analysis and the very large number of files to analyse, the tool must be able to fully use the resources of cloud computing and HPC. When running an analysis, we'll rent specific resources for a limited time, to get the analysis as fast as possible. Bear in mind that the process mustn't take more than 1 day for 100K articles. We'll rent the necessary ressources to make this possible but the tool mustn't be limited. We might use 20 CPU and 320 GB of RAM for one session, 60 CPU and 1 TB of RAM for another session and it must make the tool 3x faster in the 2nd case. When applying, please indicate your experience as for HPC platforms you might have used, and for what kind of applications.