Need to create software(script) that will be store, filter and analize data. The number of records in the database can be up to hundreds of thousands of records, you should use the most optimized algorithms and development technologies.(New data will be upload every day) The process of working has next structure: .CSV data file -> Database and text comparation analizer-> Processing macros and filters -> output in .TXT format 1 step) Loading data from a .CSV file with a fixed structure in the current database 2) After data uploaded, it should compare one field(text) for each record with another records in !ALL DATABASE for similarity (!!! This is the most difficult part of this project, it need to compare two text(two records), and return similirity of it in percents) Example of working you can find at: [login to view URL] [login to view URL] (There is russian language interface(can be translated in Google translate)) After current record was compared with all records in DB, it add info of MAXIMUM percent of similiry and ID of the record that is most similiar to. So we saved this info for each record in db. 3) One record has next structure: Field 1;Field 2;Field 3;...;Max percent of simility;ID of most similiar record 4)The ability to create flexible filters (macros) to sort the data (filters (macros) should be able to save) Macro consists of several filters (fields has different types: date, text, numerical).. For example Macro = ( Field 1 contains "John" And Field 4 is equal to "address" OR field 4 is equal to "Andy" ) So macros has a complex structure with the logical relations between the filters inside AND \ OR 5) After processing the macro data that we received, export in .TXT file !!!ALL ADDITIONAL INFO AND DATA SAMPLE WILL BE PROVIDED!!!
We are freelance software developers. If you contact me I can give a quote for your project and we can discuss the details. www.<b><i>Removed by Admin</i></b>
I have executed a similar project for a logistics company in which I had to match customer names and addresses. It required daily updation as in your case.
Matching algorithm is certainly tricky, but I have seen it getting implement across customers and it always need to be hugely modified depneding upon the requirements.
Please share the process flow diagram of the problem statement in case available.
Would like to hear back from you!
Thanks,
Rishi