Færdiggjort

[PYTHON & GCP EXPERT] Data ingestion and processing from Pub/sub to Bigquery -- 4

I'm trying to automate the ingestion of data from Pub/sub to Bigquery. I need help writing a cloud function to subscribe to a pub/sub, do some processing on that data to apply a certain logic, and then insert the data into Bigquery. I intend to use cloud scheduler, cloud pub/sub, cloud functions, and big query. Do not intend to use an apache beam as the volume of data is very low. (<100 records per day)

Deliverables

Complete writing cloud function which does the following:

1. Picks up data from pub/sub.

2. Processes the data and compares it to data in another table in BigQuery.

3. Based on certain conditions and criteria, the cloud function will insert that record into a separate table with an additional column that describes the status as 'Met', 'Not Met', 'Partially Met'

Please find project details in the attached excel file.

[login to view URL]

1. Data flow -> Pub/sub - Cloud function - Big Query

2. Flow logic - Every time data is pulled, the cloud function must use logic to map the ID, category, and status from the source table(highlighted in green) to Vul-CCF mapping, Vul-status to Control Health status, vul codename to app name with the environment and insert/update record in the destination table.

Evner: Python, Software Arkitektur, MySQL, Google Cloud Platform, Databehandling

Se mere: data entry processing officer, excel data entry processing controls, data entrydata processing, data entry processing skills, python script extract data web page, python script crape data, simply data email processing, python read input data file, scarborough data entry processing, python script extract data, python script read data text file, python script extract data website, expert data processing, python data stream processing, gcp pub sub ack, data being published to pub/sub must be sent as a bytestring., data being published to pub sub must be sent as a bytestring, google functions pub sub python

Om arbejdsgiveren:
( 48 bedømmelser ) Siem Reap, Cambodia

Projekt ID: #30987653

Tildelt til:

abhishekkarna007

Hi. I am an experienced Full stack developer with expertise in Python, GCP, Bigquery and many more. I have done a similar project in my past and am sure to complete this within reasonable time. In previous project , a Flere

$250 USD in 2 dage
(4 bedømmelser)
3.0

4 freelancere byder i gennemsnit $135 timen for dette job

SunshineConsult

Hello,I work on GCP,AWS and Python having 14 years of professional experience,can create a cloud function using python as required,Thanks.

$156 USD in 2 dage
(5 bedømmelser)
3.3
agupta2812

Hello there! Good day! I am interested in this project. I have experience working on similar projects for more than 8 years. I am an expert in Python/GCP. Please reach out to me to discuss further. Thank you!

$125 USD in 7 dage
(3 bedømmelser)
2.4
Shone15Paul

AN EXPERT IN ABOVE NAMED SKILLS, DEAR CLIENT, Thank you for sharing your project requirements. After KEENLY and PROFESSIONALLY reading your description and being able to comprehend CLEARLY with Mentioned REQUIRED SKIL Flere

$10 USD på 1 dag
(1 bedømmelse)
0.5