Notice: This project is for a FUNCTION written in PHP, not an entire website!
I Need this to be ran by calling 1 function (which may call other supporting functions if needed)
@url - determines website to crawl
@return_404 - determines if it will return errors on bad links. If this is set to true and a bad link is found, it needs to return the page the link was found on, and the bad link itself.
@use_robots_file - determines if the spider will follow the rules of the [url removed, login to view] file. If set to true, rules need to be followed exactly.
@use_meta_tags - determines if the spider will follow the spider meta tags. If set to true, rules need to be followed exactly.