Web Scraping from ~20 websites: retrieve simple information from catalogs of biological data

  • Status: Closed
  • Præmier: $300
  • Modtagne indlæg: 4
  • Vinder: rkfcccccc

Konkurrence Instruktioner

Dear all,

I am looking for a specialist in data scrapping/python who could help me retrieve information from a few websites, using a list of keywords or a uniform structure of web-pages of these websites.

In brief, there are ~20 websites that contain catalogs of germs and their optimal growth temperatures. I need to extract the name of these germs and the values of their optimal growth temperatures, then merge these data into one .csv file, remove duplicates, and make a few minor changes to this file to make it ready to use by others. This way, we can create a unified and the largest list of germs with experimentally determined growth conditions (instead of having 20 different and difficult-to-use websites). The rest are details.

How our work will be organized:

Step 1
I use this contest to choose a person who can do the work. To win this contest, you need to retrieve names of germs and their growth temperatures from these two databases and show me a random print screen (or any other evidence) of the output files:

Database A:
https://www.bcrc.firdi.org.tw/wwwbcrc/WDCM_RefStrain-tableSearch.jsp

You will see a table of germs (Trametes versicolor, Aspergillus brasiliensis, etc) in which each germ having a link to its unique web page. Here is the page that describes the first germ in the list, Trametes versicolor: https://catalog.bcrc.firdi.org.tw/BcrcContent?bid=36525

Each page of this database is organized uniformly, with germ’s names following the word “Organism:” and the optimal growth temperature value following the word “Growth conditions:”


The expected output file describing this database will be a simple .csv file with two columns:

Organism Temperature (°C)
Trametes versicolor 26
Aspergillus brasiliensis 24
Aspergillus flavus 24
and so on.



Database B:
https://www.sci.muni.cz/ccm/index.html

Here you can see a full catalog where you need to click a certain letter or number, pick a “microorganism” (=germ) to get to a page that contains the germ’s name (first two words on the page), and its optimal growth temperature (the last value on the page).

For instance, just go to the Alphabetical List of Species,
then pick the letter “A”,
and the pick the first germ in the list,
it will bring me to the page that contains the germ’s name “Granulicatella adiacens” and its growth temperature “37 °C”

As in the previous case, the output file for this database will be a simple .csv file with two columns:

Organism Temperature (°C)
Granulicatella adiacens 37
Abiotrophia defectiva 37
etc. etc. etc.


Step 2
When I see that you can deliver, we have a brief Zoom/Skype meeting to discuss the rest of the project. I award you the project and we work for a couple of days to finish everything up before I release the payment and we call it a deal.

Attached is the list of the remaining databases, so that you understand the scale and the complexity of the project. I am expecting to work together with you, brainstorming, suggesting shortcuts and solutions and accepting that some websites will be too hard to tackle.

Anbefalede Evner

Arbejdsgiverfeedback

“Egor is exceptional and highly prolific! Pleasure working with him and will definitely consider for my future projects.”

Profilbillede sergeyvmelnikov, United States.

Bedste indlæg fra denne konkurrence

Se flere indlæg

Offentlig Præciserings Opslagstavle

  • ashikmohann
    ashikmohann
    • 3 uger siden

    hi, is the contest closed?

    • 3 uger siden
  • bsharp101
    bsharp101
    • 3 uger siden

    Are entries still accepted? Thanks in advance

    • 3 uger siden
    1. sergeyvmelnikov
      Konkurrenceafholder
      • 3 uger siden

      I will clarify this point by today's 12 pm UK time as I am still figuring out if I can proceed with entry #3 .

      • 3 uger siden
    2. bsharp101
      bsharp101
      • 3 uger siden

      Ok, thanks for clarifying and waiting for your confirmation

      • 3 uger siden

Sådan kommer du i gang med konkurrencer

  • Opret din konkurrence

    Opret din konkurrence Hurtigt og nemt

  • Få tonsvis af indlæg

    Få tonsvis af indlæg Fra hele verden

  • Tildel det bedste indlæg

    Tildel det bedste indlæg Download filerne - Nemt!

Opret en Konkurrence Nu eller slut dig til os i dag!