Java/Python development work on Red Sqirl

  • Status: Closed
  • Præmier: $750
  • Modtagne indlæg: 5

Konkurrence Instruktioner

We are looking for someone with Java/Python /Docker & REST skills to do the following:

1. Add open data sources to Red Sqirl platform (see [login to view URL] for more details) according to instructions which will be provided. See here for an introduction: [login to view URL]

[ A list of open data sources will also be provided. A sample data source is: [login to view URL] ]

2. Build simple work flow to show that data source has been successfully added to Red Sqirl.

A Docker image on which you can develop and test your work can be found here: [login to view URL]

[ If you have a Hadoop cluster you can also run Red Sqirl on that.]

Anbefalede Evner

Bedste indlæg fra denne konkurrence

Se flere indlæg

Offentlig Præciserings Opslagstavle

  • aidanxc
    Konkurrenceafholder
    • 4 måneder siden

    For clarity, the winning entry will be the first to put the package on the Red Sqirl market place (http://marketplace.redsqirl.com/) where we can check that it actually works.

    • 4 måneder siden
  • kenobrien76
    kenobrien76
    • 4 måneder siden

    Yes I get the fact that the docker image includes RedSqirl and an Hadoop Cloudera image. I'm a newbie to RedSqirl, but from looking at the docs it appears to be a data analysis tool based to the hadoop ecosystem. I didn't realize it was also responsible for adding data to HDFS. I just assumed you point it to your Hadoop and perform analysis on that. So my question on scaling 'around the data ingestion' how is this handled for large datasets, do I have a way to distribute this or is it just handled with threads in the generated package

    • 4 måneder siden
    1. aidanxc
      Konkurrenceafholder
      • 4 måneder siden

      Hi Kenneth, we will get back to you on your previous question. However, just wanted to let you know that the competition has been extended for a week. The task we would like to have completed it really straightforward once you familiarise yourself with Red Sqirl.

      1. Try to get Red Sqirl to run (e.g. using Docker).
      2. Try to understand how a Red Sqirl package works (explanation on the website www.redsqirl.com)
      All actions have an input and an output

      3. Try to create your own package (sample package on the website)
      4. Once you have the sample package running just change it to use the new REST data source.
      Get the REST data and use as the action output

      • 4 måneder siden
  • megaqujik
    megaqujik
    • 5 måneder siden

    Hi,

    could you please give some hints what should be used in
    http://services.roadworks.org/sdep-publication/v5.2/sdep.asmx?op=RequestPull

    for
    Organisation fields, serviceUrl and time in DateTimeRequested="dateTime".

    • 5 måneder siden
    1. aidanxc
      Konkurrenceafholder
      • 4 måneder siden

      SOME ADVICE

      1. Try to get Red Sqirl to run (e.g. using Docker).
      2. Try to understand how a Red Sqirl package works (explanation on the website www.redsqirl.com)
      All actions have an input and an output

      3. Try to create your own package (sample package on the website)
      4. Once you have the sample package running just change it to use the new REST data source.
      Get the REST data and use as the action output

      • 4 måneder siden
  • rukkykf
    rukkykf
    • 4 måneder siden

    is there any other way to get the image file. The docker pull request isn't going through. I've tried several times now, it's impossible for me to get the file because of my slow connection. However, if you have another way to distribute the image, perhaps as a downloadable tar file that I can download with a download manager (not docker), that would be nice.

    • 4 måneder siden
    1. rukkykf
      rukkykf
      • 4 måneder siden

      to be clearer: it will be easier for me to download the redsqirl image as a tar file and load it using docker load. You can create a tar file with docker save and host the tar file on a file sharing service like dropbox or google drive.

      • 4 måneder siden
    2. aidanxc
      Konkurrenceafholder
      • 4 måneder siden

      SOME ADVICE

      1. Try to get Red Sqirl to run (e.g. using Docker).
      2. Try to understand how a Red Sqirl package works (explanation on the website www.redsqirl.com)
      All actions have an input and an output

      3. Try to create your own package (sample package on the website)
      4. Once you have the sample package running just change it to use the new REST data source.
      Get the REST data and use as the action output

      • 4 måneder siden
  • karthiks416
    karthiks416
    • 4 måneder siden

    Hi, Can you provide any other documentation. The provided link doesn't have much information like how to add REST and how to add data source

    • 4 måneder siden
    1. aidanxc
      Konkurrenceafholder
      • 4 måneder siden

      SOME ADVICE

      1. Try to get Red Sqirl to run (e.g. using Docker).
      2. Try to understand how a Red Sqirl package works (explanation on the website www.redsqirl.com)
      All actions have an input and an output

      3. Try to create your own package (sample package on the website)
      4. Once you have the sample package running just change it to use the new REST data source.
      Get the REST data and use as the action output

      • 4 måneder siden
  • aidanxc
    Konkurrenceafholder
    • 4 måneder siden

    SOME ADVICE

    1. Try to get Red Sqirl to run (e.g. using Docker).
    2. Try to understand how a Red Sqirl package works (explanation on the website www.redsqirl.com)
    All actions have an input and an output

    3. Try to create your own package (sample package on the website)
    4. Once you have the sample package running just change it to use the new REST data source.
    Get the REST data and use as the action output

    • 4 måneder siden
  • aidanxc
    Konkurrenceafholder
    • 4 måneder siden

    This is a relatively simple piece of work.

    • 4 måneder siden
  • aidanxc
    Konkurrenceafholder
    • 4 måneder siden

    The competition has been extended for another week.

    • 4 måneder siden
  • aidanxc
    Konkurrenceafholder
    • 4 måneder siden

    Thank you for trying.

    • 4 måneder siden
  • aidanxc
    Konkurrenceafholder
    • 4 måneder siden

    Nobody managed to complete the task. We will review the details of this contest and may repost it with a simpler objective.

    • 4 måneder siden
  • kenobrien76
    kenobrien76
    • 4 måneder siden

    Hi. Have a couple of questions on this. What are the scaling requirements for this data ingestion application? Do you need live metrics as data is being added? Do you need an interface to add/update the data ingestion jobs? Where do you plan on executing the application (cloud, local server etc)? Thanks,

    • 4 måneder siden
    1. aidanxc
      Konkurrenceafholder
      • 4 måneder siden

      RedSqirl is inside the docker image. The docker is based on Cloudera image!
      you need create the package and run inside docker! The package should get the data using the Rest API and this should be the output of the action.
      Just follow the website instructions for create a package! Than you just need add the Rest call there.

      Here: https://redsqirl.com/pckdev.html

      • 4 måneder siden
  • kenobrien76
    kenobrien76
    • 4 måneder siden

    Also have you consider something like spring cloud data flow https://cloud.spring.io/spring-cloud-dataflow/ or is a custom built application a requirement?

    • 4 måneder siden
  • rukkykf
    rukkykf
    • 5 måneder siden

    Do you have a sample open data source we can use for this contest that uses REST? The one you put in the contest description from the RoadWorks live API uses SOAP, not REST. I'm happy to proceed adding the data source from the RoadWorks API, but if you have another sample I could use instead, that'd be great.

    • 5 måneder siden
    1. aidanxc
      Konkurrenceafholder
      • 5 måneder siden

      You can try with any REST API on the website you can filter for API
      here is on example https://api.oireachtas.ie

      • 5 måneder siden
  • rukkykf
    rukkykf
    • 5 måneder siden

    There are some other open data sources from the data.gov.uk site that use REST, should I proceed with one of those?

    • 5 måneder siden
    1. aidanxc
      Konkurrenceafholder
      • 5 måneder siden

      Yes, please go ahead with any other data source that you like.

      • 5 måneder siden
  • ITPyramid85
    ITPyramid85
    • 5 måneder siden

    Hi, Sir. Can I implement your project on Offline Red Sqirl?

    • 5 måneder siden
    1. aidanxc
      Konkurrenceafholder
      • 5 måneder siden

      You can do it on Docker if you download it from the link provided.

      • 5 måneder siden
    2. aidanxc
      Konkurrenceafholder
      • 5 måneder siden

      Hi there, were you able to use Docker?

      • 5 måneder siden
  • aidanxc
    Konkurrenceafholder
    • 5 måneder siden

    Hi there, there is some flexibility in that regard. What we want to assess is your ability to add a (data) package to Red Sqirl (based on a publicly available API). .

    For the API in question (i.e. https://data.gov.uk/dataset/1d9d6499-6ff5-4450-839a-68023c58452d/live-roadworks-api)

    We would be happy if you could bring in the location, start date/time, expected end date/time and description.

    • 5 måneder siden

Vis flere kommentarer

Sådan kommer du i gang med konkurrencer

  • Opret din konkurrence

    Opret din konkurrence Hurtigt og nemt

  • Få tonsvis af indlæg

    Få tonsvis af indlæg Fra hele verden

  • Tildel det bedste indlæg

    Tildel det bedste indlæg Download filerne - Nemt!

Opret en Konkurrence Nu eller slut dig til os i dag!