We are looking for a professional with experience in crawling sites and social networks or using available API’s where possible.
The project will be based on the following networks:
There will be two different ways of crawling or accessing the information:
- crawling upon request to build up information based on a keyword that can be found in user name, user comments, profile, tweets or videos/pictures or by a user name
- updating present information
All the data need to be stored in a database (data warehouse solution).
The information we need are as follows and might go beyond what is being accessible by API and thus would need the implementation of a crawler:
- user name of profile that matches search criteria
- number of friends to that profile
- name of friends
- every information that is possible to get about the person: sex, age, hometown, place of stay
- for friends interactivity: who is commenting most to user’s wall and to which friends does the user comment the most
- uploaded videos, pictures and wall entries that match search criteria
- username plus any information available about the persons as sex, age etc.
- subscribed tweets (need to be scanned for search criteria as well)
- number and list of subscriber
- subscriber’s interactivity: which followers are commenting the most and on which tweets does the user comment the most
- username, number of friends, list of friends
- list of uploaded videos and comments
- crawling comments and video replies for search criteria and save matching entries
- friend’s interactivity: which friends comment the most and on which friends does the user comment the most
At this point we are open for suggestions which programming language should be used.