ADR Web Scraping Tool

Project Overview We are looking for a solution to deploy a job search website based on the job listing data contained on other sites. We envision this solution being developed in three distinct components, a spidering/scraping utility, the administrative interface and the front facing web interface. NOTE: The requirements for this job are listed below and are included in the attached file which has copies of all diagrams which would not copy. This project description together with the attached file comprise the complete description and requirements for this project. Spidering/Scraping Utility This utility will be able to: 1. Spider the job site based on the supplied URL, scan listings for relevant keywords, retrieve the matching text of the job posting along with the job posting title 2. Store a local copy of the retrieved job listing in a persistent storage method (database/XML files) 3. Run as an automated job (cron) on the server on a nightly basis 4. Be able to purge old job listings automatically 5. Create a full-text index of the retrieved job listings Administrative Interface The administrative interface will be used by the business owner/user to manage the job sites and the associated job listings retrieved. This interface must: 1. Allow for the entry of a base URL of a job site, keywords to search 2. Be able to be configured by a business user 3. Be password protected/require authentication 4. Allow for the modification or deletion of the retrieved job listings Administrative Sample Screenshots Front Facing Website The front facing website component of this project must provide the following: 1. A keyword searchable interface 2. After search return a list of job titles in order of relevance 3. Allow the user to set how many results are displayed on the page, 10, 20, 50, 100 4. The site will be embedded/iFramed into an existing site so no additional branding, navigation or other embellishments are needed. Front Facing Sample Screenshot Applicable stories Initial attempt at stories. Done in the form: As a <<who>> I want to <<what>> so that <<why>> 1. As a business user I want to enter a job site URL, a series of keywords for jobs to be pulled so that the site will be spidered and retrieve relevant listings. 2. As end users I want to be able to search jobs and receive relevant listings High Level Design Technical/Architectural Considerations Proposed solutions: - may take advantage of existing technologies such as WordPress - should use open source where possible - not use any proprietary software code - solution should be a utility that can be used to scrape any site NOT a one off per site model. Implementation/Deployment Guidance The solution will be deployed by the vendor at a hosting site of the customer’s choosing. This includes the configuration of the database, installation of the software/scripts and any web configuration. All user names and passwords will be sent to the customer after deployment is completed. The final product after purchase will be the property of the customer. Example Sites These are provided as a representative sampling only. Some or all of these sites and/or additional sites may be used at times while deploying the solution. [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] [login to view URL] http://www.monster.c