Airflow , Kafka Kinesis Project

$8-15 USD / hour

Imefungwa

Imechapishwa

over 3 years ago

$8-15 USD / hour

A>Airflow 1. Take any small source CSV file and copy it into target S3 using Airflow 2. Take any CSV in S3 and copy that into target local server using airflow 3. On local server, airflow calls a simple python file , that reads say a json and pushes data into output say CSV schedule above one after anotther to run every ten minutes ( since the files are small, we can run them back to back) B> Kafka (and / or) Kinesis 1. Take any Kafka producer code that writes into topic , pyspark consumer reads from topic 2. Say the kafka producer has hundred records. consumer reads say thirty records and goes down. Now the kafka consumer should be restarted from thirty first record. This is part of off-set management. Checkpoints to be created. feel free to take up PartA> Airflow or PartB> Kafka &/Kinesis ,separately based on your preference.

Python

Software Architecture

Linux

Data Processing

ETL

Kitambulisho cha mradi: 28252219

Kuhusu mradi

4 mapendekezo

Mradi wa mbali

Inatumika 3 yrs ago

Unatafuta kupata pesa?

Barua pepe

Faida za kutoa zabuni kwenye Freelancer

Weka bajeti yako na muda uliopangwa

Pata malipo kwa kazi yako

Eleza pendekezo lako

Ni bure kujiandikisha na kutoa zabuni kwa kazi

4 wafanyakazi huru wana zabuni kwa wastani $18 USD/saa kwa kazi hii

@maadirajudata

Hi I am interested in part b Kafka producer and consumer. Hope it can be achieved in local environment please confirm I have expertise in Kafka based applications thanks maadiraju

$22 USD ndani ya siku 20

4.9

(31 hakiki)

5.5

@anzarulislam

Hello, I want to do your Airflow , Kafka Kinesis Project But I have few questions. Can I ask? Let's talk and start now. Best, Anzarul.

$15 USD ndani ya siku 40

4.7

(19 hakiki)

5.1

@DevraniAnuj

Hi I have worked with Airflow extensively. I am interested in the A part. Please explain in detail so that we can get started quickly.

$17 USD ndani ya siku 10

0.0

(0 hakiki)

0.0

@bharatthe1

Greetings, How are you! After going through your requirements, I am interested in Part A and I would like to share that I can fulfill your requirement. Below is brief description about my self. I am having 8+ years of working experience in Technologies like Python, Java and other big data technologies like Spark, Hive and Hadoop. Also developed many spark applications in AWS by implementing AWS Services like Lambda, Step Functions, SNS, Glue, Redshift. Also I have created Airflow DAGs by creating custom hooks and operators to connect to s3, Lambda, EMR, and SSM(System Manager) and deployed these schedulers in our prod implementation. With above experience and skillset, I strongly feel that you will be highly satisfied with the quality of work and on time delivery. You can reach out to me any time. Looking forward to hear from you! Thanks.

$18 USD ndani ya siku 40