Im in need of a script which crawls rss feeds...
I have all the rss urls in a database, i have tryed a program called
Magpie RSS - PHP RSS Parser ([url removed, login to view]) but it cant crawl certain rss url like ( [url removed, login to view] ).
So what does the system need?
1. Should be able to crawl atleast 95% of the url i have in my database, each rss feed is linked to a user id.
2. It should rank rss feeds depending on how often they type new articles, so the crawl script doesnt crawl a rss feed everyday if the writer only types once a month, makes sence?.
3. If it cant crawl a url it should jump over it and try a new one.
4. The script should crawl a maximum of 10 rss feeds each time the script runs. I will set it in cronjobs to run every 5 minutes.
5. Your allowed to use the magpie script if you mod it to work with 95% of the urls.
6. i will send you a txt message with all the urls then its up to you to create a database for it since i cant give you access to a live website.
7. it should crawl (link to the article) (title) (the article (1000 letters max))and the date/time.