Java/Python development work on Red Sqirl

  • Status: Closed
  • Præmier: $750
  • Modtagne indlæg: 6

Konkurrence Instruktioner

We are looking for someone with Java/Python /Docker & REST skills to do the following:

1. Add open data sources to Red Sqirl platform (see [login to view URL] for more details) according to instructions which will be provided. See here for an introduction: [login to view URL]

[ A list of open data sources will also be provided. A sample data source is: [login to view URL] ]

2. Build simple work flow to show that data source has been successfully added to Red Sqirl.

A Docker image on which you can develop and test your work can be found here: [login to view URL]

[ If you have a Hadoop cluster you can also run Red Sqirl on that.]

Anbefalede Evner

Bedste indlæg fra denne konkurrence

Se flere indlæg

Offentlig Præciserings Opslagstavle

  • aidanxc
    Konkurrenceafholder
    • 2 months ago

    For clarity, the winning entry will be the first to put the package on the Red Sqirl market place (http://marketplace.redsqirl.com/) where we can check that it actually works.

    • 2 months ago
  • kenobrien76
    kenobrien76
    • 3 months ago

    Yes I get the fact that the docker image includes RedSqirl and an Hadoop Cloudera image. I'm a newbie to RedSqirl, but from looking at the docs it appears to be a data analysis tool based to the hadoop ecosystem. I didn't realize it was also responsible for adding data to HDFS. I just assumed you point it to your Hadoop and perform analysis on that. So my question on scaling 'around the data ingestion' how is this handled for large datasets, do I have a way to distribute this or is it just handled with threads in the generated package

    • 3 months ago
    1. aidanxc
      Konkurrenceafholder
      • 2 months ago

      Hi Kenneth, we will get back to you on your previous question. However, just wanted to let you know that the competition has been extended for a week. The task we would like to have completed it really straightforward once you familiarise yourself with Red Sqirl.

      1. Try to get Red Sqirl to run (e.g. using Docker).
      2. Try to understand how a Red Sqirl package works (explanation on the website www.redsqirl.com)
      All actions have an input and an output

      3. Try to create your own package (sample package on the website)
      4. Once you have the sample package running just change it to use the new REST data source.
      Get the REST data and use as the action output

      • 2 months ago
  • megaqujik
    megaqujik
    • 3 months ago

    Hi,

    could you please give some hints what should be used in
    http://services.roadworks.org/sdep-publication/v5.2/sdep.asmx?op=RequestPull

    for
    Organisation fields, serviceUrl and time in DateTimeRequested="dateTime".

    • 3 months ago
    1. aidanxc
      Konkurrenceafholder
      • 2 months ago

      SOME ADVICE

      1. Try to get Red Sqirl to run (e.g. using Docker).
      2. Try to understand how a Red Sqirl package works (explanation on the website www.redsqirl.com)
      All actions have an input and an output

      3. Try to create your own package (sample package on the website)
      4. Once you have the sample package running just change it to use the new REST data source.
      Get the REST data and use as the action output

      • 2 months ago
  • rukkykf
    rukkykf
    • 3 months ago

    is there any other way to get the image file. The docker pull request isn't going through. I've tried several times now, it's impossible for me to get the file because of my slow connection. However, if you have another way to distribute the image, perhaps as a downloadable tar file that I can download with a download manager (not docker), that would be nice.

    • 3 months ago
    1. rukkykf
      rukkykf
      • 3 months ago

      to be clearer: it will be easier for me to download the redsqirl image as a tar file and load it using docker load. You can create a tar file with docker save and host the tar file on a file sharing service like dropbox or google drive.

      • 3 months ago
    2. aidanxc
      Konkurrenceafholder
      • 2 months ago

      SOME ADVICE

      1. Try to get Red Sqirl to run (e.g. using Docker).
      2. Try to understand how a Red Sqirl package works (explanation on the website www.redsqirl.com)
      All actions have an input and an output

      3. Try to create your own package (sample package on the website)
      4. Once you have the sample package running just change it to use the new REST data source.
      Get the REST data and use as the action output

      • 2 months ago
  • karthiks416
    karthiks416
    • 2 months ago

    Hi, Can you provide any other documentation. The provided link doesn't have much information like how to add REST and how to add data source

    • 2 months ago
    1. aidanxc
      Konkurrenceafholder
      • 2 months ago

      SOME ADVICE

      1. Try to get Red Sqirl to run (e.g. using Docker).
      2. Try to understand how a Red Sqirl package works (explanation on the website www.redsqirl.com)
      All actions have an input and an output

      3. Try to create your own package (sample package on the website)
      4. Once you have the sample package running just change it to use the new REST data source.
      Get the REST data and use as the action output

      • 2 months ago
  • aidanxc
    Konkurrenceafholder
    • 2 months ago

    SOME ADVICE

    1. Try to get Red Sqirl to run (e.g. using Docker).
    2. Try to understand how a Red Sqirl package works (explanation on the website www.redsqirl.com)
    All actions have an input and an output

    3. Try to create your own package (sample package on the website)
    4. Once you have the sample package running just change it to use the new REST data source.
    Get the REST data and use as the action output

    • 2 months ago
  • aidanxc
    Konkurrenceafholder
    • 2 months ago

    This is a relatively simple piece of work.

    • 2 months ago
  • aidanxc
    Konkurrenceafholder
    • 2 months ago

    The competition has been extended for another week.

    • 2 months ago
  • aidanxc
    Konkurrenceafholder
    • 2 months ago

    Thank you for trying.

    • 2 months ago
  • aidanxc
    Konkurrenceafholder
    • 2 months ago

    Nobody managed to complete the task. We will review the details of this contest and may repost it with a simpler objective.

    • 2 months ago
  • kenobrien76
    kenobrien76
    • 3 months ago

    Hi. Have a couple of questions on this. What are the scaling requirements for this data ingestion application? Do you need live metrics as data is being added? Do you need an interface to add/update the data ingestion jobs? Where do you plan on executing the application (cloud, local server etc)? Thanks,

    • 3 months ago
    1. aidanxc
      Konkurrenceafholder
      • 3 months ago

      RedSqirl is inside the docker image. The docker is based on Cloudera image!
      you need create the package and run inside docker! The package should get the data using the Rest API and this should be the output of the action.
      Just follow the website instructions for create a package! Than you just need add the Rest call there.

      Here: https://redsqirl.com/pckdev.html

      • 3 months ago
  • kenobrien76
    kenobrien76
    • 3 months ago

    Also have you consider something like spring cloud data flow https://cloud.spring.io/spring-cloud-dataflow/ or is a custom built application a requirement?

    • 3 months ago
  • rukkykf
    rukkykf
    • 3 months ago

    Do you have a sample open data source we can use for this contest that uses REST? The one you put in the contest description from the RoadWorks live API uses SOAP, not REST. I'm happy to proceed adding the data source from the RoadWorks API, but if you have another sample I could use instead, that'd be great.

    • 3 months ago
    1. aidanxc
      Konkurrenceafholder
      • 3 months ago

      You can try with any REST API on the website you can filter for API
      here is on example https://api.oireachtas.ie

      • 3 months ago
  • rukkykf
    rukkykf
    • 3 months ago

    There are some other open data sources from the data.gov.uk site that use REST, should I proceed with one of those?

    • 3 months ago
    1. aidanxc
      Konkurrenceafholder
      • 3 months ago

      Yes, please go ahead with any other data source that you like.

      • 3 months ago
  • ITPyramid85
    ITPyramid85
    • 3 months ago

    Hi, Sir. Can I implement your project on Offline Red Sqirl?

    • 3 months ago
    1. aidanxc
      Konkurrenceafholder
      • 3 months ago

      You can do it on Docker if you download it from the link provided.

      • 3 months ago
    2. aidanxc
      Konkurrenceafholder
      • 3 months ago

      Hi there, were you able to use Docker?

      • 3 months ago
  • aidanxc
    Konkurrenceafholder
    • 3 months ago

    Hi there, there is some flexibility in that regard. What we want to assess is your ability to add a (data) package to Red Sqirl (based on a publicly available API). .

    For the API in question (i.e. https://data.gov.uk/dataset/1d9d6499-6ff5-4450-839a-68023c58452d/live-roadworks-api)

    We would be happy if you could bring in the location, start date/time, expected end date/time and description.

    • 3 months ago

Vis flere kommentarer

Sådan kommer du i gang med konkurrencer

  • Opret din konkurrence

    Opret din konkurrence Hurtigt og nemt

  • Få tonsvis af indlæg

    Få tonsvis af indlæg Fra hele verden

  • Tildel det bedste indlæg

    Tildel det bedste indlæg Download filerne - Nemt!

Opret en Konkurrence Nu eller slut dig til os i dag!