Creation of a batch job that checks a website for a new file.
Need someone who can automate the download of a csv file from the server (File is about 3 GB) File has 25 Million + records
We have working Java code and Spring boot APIs that
1.) Chunks this file into multiple csv's as per the parameter
2.) We generate an automated list of partitions which partition 2 Million records in each partition
These files then need to be uploaded into a MySQL db on AWS
Table columns are already mapped so no transformation is needed.
Currently the loading part is using a My SQL INLOAD command. which we execute multiple times to load all data into the server.
<TODO 2> So we want to automate the data load into the server.
<TODO 3> On successful data load into the server we want a success notification (email) highlighting how many total records/successful/failures etc were there.
A lot of the code already exists and we have detailed instructions on running this.
If we run the instructions manually it takes about 2-2.5 hours every month to execute the whole thing but we want to automate this.
I do not need partial automation as we already have a solution - So only apply if you know Java file and DB adapters
Max budget 50$
Anyone applying will have to exhibit ample experience with this task... don't ask to discuss the project unless you have detailed what needs to be done.
Include PDI Rocks!!!! in your response to showcase you read the post - All entries without that will be deleted.
14 freelancere byder i gennemsnit $198 timen for dette job
Hello. PDI Rocks!!!! I was watching the project. If you have the instructions, there would be no problem creating a program that runs daily and does all the work automatically. Greetings
i think that si a good job for Spring Batch and create dime steps flor the automatic process , i hace experienc whith data migrate , maybe i can help u
PDI Rocks!!!! Hi I am very interested in your job and have very good skills. I understand your requirement and will complete your job PERFECTLY. I'll do my best for you. Best Regards.
I have 5 years experience in software development. I am working on last 3 years on file parsing project. I am not sure how to di it. But I will definitely do it.