Scrapy, find and fix a bottleneck: low throughput


The scraper is based on Scrapy and it's almost complete, we are scraping products from AMZ (Italy at the moment) but we are struggling with overall requests throughout.

We are using backconnect rotating proxies from 2 vendors and we have about 150 threads but we are able to scrape only 1/2 URLs per second, in addition we can use more threads/proxies with TOR but the speed is the same.

Evner: Web Skrabning, Python, Software Arkitektur, Datasøgning, Scrapy

Se mere: php read html fix mistakes, read display shapefile use java, clipshare description fix, description fix joomla, php bug fix job description, read description, read description match, javascript read description, read description match photos, fix wordpress google site description, python read description file, read description browsergame, product read description write objective, hotel description english use, read description bidding, read description twitter, tbulit 3 features in core php website budget is 60 fixed read description, tbulit 3 features in core php website budget is 60$ fixed read description

Om arbejdsgiveren:
( 12 bedømmelser ) Cagli, Italy

Projekt ID: #21115494

8 freelancere byder i gennemsnit €11/time for dette job


Hi there, I may give you some suggestion about proxy. I have 4+ years’ experience at Web scraping. If you'd like to view my previous work, take a look at my profile or check some de Flere

€12 EUR / time
(89 bedømmelser)

Hi, Client!! @$@$@$PLEASE CONTACT ME. I CAN DO IT WONDERFULLY.$@$@$@ ##% Your idea is attracted me to your world.%## I am Python expert. i have various skills. @@PLEASE GIVE ME CHANCE TO PROVE MY SKILLS@@ :-):-):-) Flere

€15 EUR / time
(61 bedømmelser)

Hello There Good Day! I've carefully read your project description. I am very interested in doing this project. I am a professional Scraping expert. I will give 100% guaranteed satisfaction from my past experience Flere

€9 EUR / time
(19 bedømmelser)

Hello, I am a professional web scraper with more than 6 years of experience and have completed many scraping tasks successfully with client satisfaction I can scrape any kind of website for you and can output data i Flere

€12 EUR / time
(7 bedømmelser)

Dear Sir. Glad to meet you. I'm Web developer specializing in web scraping crawling and indexing web pages, data entry. Skills: python, scrapy, selenium, requests, beautifulsoup, mechanize, lxml, urllib2, automation, Flere

€9 EUR / time
(10 bedømmelser)

⭐LOW RATE! HIGH QUALITY&SPEED!⭐ I read your project . Your project is very interesting. I have ever been similar project with your project. I have already finished many project like yours and I have enough experience i Flere

€9 EUR / time
(3 bedømmelser)

Hi there Hope this finds you Hale and Hearty I am a python developer, having more than 5 years of experience in Machine Learning, Deep Learning, and Data Science engineering. With my good experience in core python, I c Flere

€10 EUR / time
(2 bedømmelser)

Hi. Please survey my reviews. I saw your description carefully and I am confident I can full fill your requirement. So I provide my skills in hopes of working with you. I am full-stack developer and I have 2+ years Flere

€10 EUR / time
(4 bedømmelser)