Lukket

spark application on google cloud platform for fetching and processing data from hdfs

Hello we are looking for a scala developer who has experience working on handling data in .packet form on spark clusters on google cloud platform. Basically the task is to access data from hdfs in .packet form, query through the data for relevant UIDs, fetch some specific fields in those UIDs, process parameters by performing some mathematical computations on those fields for those specific UIDs and store the processed values in a separate .packet file on hdfs. Further aggregation needs to be performed on the computed values, and final summary file needs to be stored into Mongo dB.

The technologies you need to be comfortable with : Dataproc on google (cloud native hadoop and spark), airflow (will be used for scheduling), google cloud platform (in general), scala (for scripts), Mongo dB (for data export)

Evner: Google App Engine, Hadoop, NoSQL Couch & Mongo, Scala, Spark

Se mere: cron php google cloud platform, find developer for google cloud platform, Google Cloud Platform, hadoop on google cloud platform, no filesystem for scheme: gs, google dataproc hdfs, gcs connector maven, google cloud storage vs hdfs, gsutil hdfs, class com.google.cloud.hadoop.fs.gcs.googlehadoopfilesystem not found, google dataproc tutorial, google cloud application, google cloud print java application, web application processing data report, geofencing application google map, integrate net application google calendar, google maps info window mysql data, application google maps, google map api filter xml data source, developing facebook application google maps

Om arbejdsgiveren:
( 80 bedømmelser ) BANGALORE, India

Projekt ID: #16298138

8 freelancere byder i gennemsnit ₹12156 på dette job

lokeshyadav0005

Hello I have extensive experience working with various data formats and using Spark to deal with those. I believe I'll be able to complete this task successfully. About me: - 3 years of experience working in the f Flere

₹35000 INR in 7 dage
(6 bedømmelser)
3.7
gkbhardwaj87

I have work experience 5 year in big data technology . I have experience in elasticsearch, java, scala , spark. for more info ping m.e

₹11111 INR in 7 dage
(3 bedømmelser)
4.1
AmolZinjadeP

Hi, I have more than 3+ years of experience in Hadoop technologies like mapreduce , spark, hdfs etc. I can complete your project contact me for more details

₹10000 INR in 3 dage
(4 bedømmelser)
3.6
haadfreelancing

I am interested to work on this project as I have relevant experience in Big Data,Sqoop, Hadoop, Spark, Hive, Kafka, Spark Streaming, Rdd, Datframe, Dataset , Python, Scala, google cloud, azure, aws. I am well versed i Flere

₹11111 INR in 6 dage
(4 bedømmelser)
0.8
₹12222 INR in 3 dage
(1 bedømmelse)
0.2
dineshrajputit

hi, I am hadoop, sparkand nosql engineer with 6 years of experience. can do this, comfortable with spark, scala, airflow, Google cloude. data proc I can manage.

₹2250 INR på 1 dag
(1 bedømmelse)
0.0
khannanav

We have 8 years of experience working in Machine Learning. We have built various recommendation engines, web apps, crawlers, analytical dashboards etc. We have rich experience in Python, Spark, R, Scala, Cassandra, Hiv Flere

₹7777 INR in 3 dage
(0 bedømmelser)
0.0
bigdatabear

Languages: JAVA. Java/J2EE: Core JAVA,JAVAFX, Advanced JAVA, Servlet, JSP, JSTL, EJB, JDBC, Junit, Web Services, XML, XSD, JAX-RS, DOM, SAX, Multithreading, JTA, Custom Tags, JPA API’s. Web Technologies: Html, DHTML Flere

₹7777 INR in 3 dage
(0 bedømmelser)
0.0