We have a big list of domains we need to check and get an activity score for. To do this our thought was to attempt to get the [login to view URL] file and then parse it to get a last modified date. then we would do some basic math on this to come up with a score.
Assuming that the pages are edited in groups of dates i think we can dump in to excel a month and count so we have something like
1/2019 - 10
2/2019 - 37
3/2019 - 0
so in the excel output we have
domain | http response | found sitemap | page count from sitemap | months for the last 24.... (make a column for each month for the last 24 months)
one thing to make sure you got in the sitemap is that sometimes sitemaps are nested. you will need to follow the link if it leads to another sitemap for that section. for example using google [login to view URL] this main map links to sub maps.
if the domain redirects or fails we want to log that I think we can log the http response codes for this so something like
2xx - fine
4xx - failed
5xx - failed
3xx- redirect, then log the redirect name it sends back
app should run on windows. and would be cool if we can have config file to put the path to the csv of domains, and maybe some thread count so we can set the performance characteristic of this. We will be running this on some big lists of domains so it might be good to make sure its able to handle running a big list like 10K in some controlled threaded fashion. also we should think about a timeout after a minute or something reasonable so the app doesnt freeze for a dead or bad url?
I found this lib which might help you if you needed it - [login to view URL]
29 freelancere byder i gennemsnit $301 på dette job
I can write python app for Windows with GUI using the mentioned python lib and multi threading . It will save results in csv as per description . I have 6+ experience in Python .
Hi there. Thank you for your posting. I have read your posting carefully and I would like to work with you. I hope we can have a detailed discussion by chat and share our idea. Regards.
Hi! there. ⭐Thanks⭐ good post. I am web scrapper. I am ready in any language such as PHP, Python. I have rich experience in scraping many sites. I want to meet you on the chatting. Regards.
I can start your project immediately. I can provide full-time communication and work your time-zone. If you give me a chance to serve you, I will provide a high quality product within the deadline. Best Regards