We have a simple python collector that scrapes data from a public web site.
It primarily uses one URL (with changing object IDs and date parameters).
In some cases, it requires drilling down to a second level page.
The collection will probably only make 300 - 500 requests a week.
The technical requirements will generally be:
1 -- you will read from Rabbit MQ
2 -- you will write to Kafka (raw non-transformed data).
3 -- you will provide a docker file and config using env vars. Our Collections team can provide the standard set of config vars so we can inject it into our standard run setup.
4 -- you will manage your retry and error logic -- we will give you a logging format.
Our team is fully booked and cannot get to this small collection but will need to support it.
More details after we see a few responses.
20 freelancere byder i gennemsnit $18/timen for dette job
Hi, I am Python script developer with 10 years of experience. I can scrape required website by python script/bot with your instructions very short time. Can we discuss please? Thanks.
So you want a dockerized version of a scrapper? A pretty boring job. The third party may get frustrated and ban bots access. What do you make with these data? Let's research some REST APIs.