Web scraper in Python with Scrapy for Forums Sites

I have several forums that i need scrapped this job is for the first one a xeno forum you can see the full description here See attached document for more detail. Example Site: [login to view URL] Sitemap: The have a robust sitemap may be the easiest way to go. Your choice though not a requirement [login to view URL] Phases 1. MVP Extract small set of records of what will be the final deliverable this includes Categories, , topics, posts, users,images.... The more data the better 2. Full Extract Provide Script Provide data set - All current Records - can be massive json file or mongo db database. I can provide database or you can send me a docker image 3. Differential Extract Job Provide Script job will have a start and end time frame so i can run it daily or for a date range for example 1 week. Same result as above but subset based on time. Extracted Data Example Here is the starting of the extract. I would like to get as much metadata. This list is not a final list and the names of the properties are just for demo purpose and can be changed. Final naming will be determined based on the first mvp delivery. 1. Categorys ■ ID ■ title ■ name ■ forum ■ description ■ url ■ countTopics ■ countPosts ■ lastPostDate ■ lastPostUser{} 2. Topics ■ ID ■ parentCateogry_id ■ name ■ url ■ dateCreate ■ userID ■ IsPinded ■ isClosed ■ isRedirect ■ LastPostDate ■ LastPostUser ■ images[] 3. Posts ■ ID ■ userID ■ topicID ■ categoryID ■ Url ■ Content ■ Content_html ■ dateCreate ■ Likes ■ isTopicStart ■ images[] 4. Users ■ ID ■ URL ■ imageURL ■ dateLastActive ■ dateJoined ■ messagesCount ■ likesCount ■ galleryCount ■ Articles ■ Birthday ■ Location ■ Badges[] Below is an example topic and posts page with some overlays of the data sets [login to view URL] Extra: Images: I would like download all the images as well. Images would include but are not limited to images in the post and user profile images. Should have user’s image and the images in the topics/posts in separate folder. Example Possible jSON Extracts in attached file