Lead Data Engineer - Kafka

Role overview

Lead Data Engineer - Kafka at Relanto

Real-Time Data Engineering Design, develop, and maintain real-time data pipelines using Apache Flink and Apache Kafka.

Responsibilities

• Develop production-grade streaming applications for high-volume and low-latency workloads.
• Implement data transformation, filtering, enrichment, aggregation, and event processing.
• Build reliable event-processing pipelines with appropriate error handling and recovery mechanisms.
• Consume and publish events across Kafka topics.
• Implement appropriate partitioning, consumer groups, offsets, and delivery mechanisms.
• Troubleshoot streaming pipeline failures and performance issues.
• Apache Flink Develop and maintain Apache Flink jobs forreal-time data processing.
• Implement: Stream transformations Filtering Mapping Aggregations Joins Windows Event-time processing Watermarks State management Implement Flink checkpointing and recovery mechanisms.
• Optimize Flink jobs for performance, scalability, and resource utilization.
• Monitor Flink jobs for latency, throughput, failures, and resource consumption.
• Troubleshoot state, checkpointing, backpressure, and processing issues.
• Kafka Develop Kafka-based ingestion and streaming pipelines.
• Create and manage Kafka topics and event streams.
• Workwith partitions, offsets,consumer groups, replication, and retention.
• Implement reliableproducer and consumer applications.
• Handle message ordering, retries,duplicate events, and replay scenarios.
• Monitor Kafka performance and troubleshoot consumerlag and throughput issues.
• Work with Kafka schemasand serialization formats.
• CDC & Debezium BuildCDC-based ingestion pipelines using Debezium .
• Configure and maintain Debeziumconnectors.
• Capture source-system inserts, updates,and deletes.
• Publish CDC events into Kafka.
• Handle initial snapshots and incremental CDC processing.
• Manage schemaevolution and changesin source systems.
• Implement data reconciliation and consistency checks.
• Troubleshoot CDC failures and source-to-target data issues.
• Data Orchestration Develop and maintain data workflows using Apache Airflow orequivalent orchestration frameworks.
• Buildreusable DAGs for: Data ingestion CDC workflows Data validation Flink job execution Data transformation ClickHouse loading Downstream integrations Implement workflowdependencies, scheduling, retries,backfills, SLAs, and alerting.
• Integrate Airflow workflows with Kafka, Flink, Debezium, ClickHouse, APIs, and cloud services.
• Monitor workflow execution and troubleshoot failures.
• Develop reusable operators, sensors,and workflow components where appropriate.
• Useevent-driven triggers where real-time workflows require them.
• ClickHouse & Analytical Data Integrate streaming data pipelines with ClickHouse .
• Design efficient analytical data models.
• Develop and optimize SQL queries.
• Implement appropriate partitioning, sorting, indexing,and retention strategies.
• Optimize data ingestion and query performance.
• Support downstream analytical use cases,dashboards, and reporting requirements.
• Data Quality& Reliability Implement data validation and quality checksthroughout the pipeline.
• Buildreconciliation mechanisms betweensource and target systems.
• Monitor data freshness, completeness, accuracy, and consistency.
• Implement error handling, retry,replay, and recoverymechanisms.
• Establish loggingand observability for critical pipelines.
• Support incident investigation and root-cause analysis.
• Integration & APIs Integrate streaming and analytical data with APIs, endpoints, dashboards, and downstream applications.
• Develop data interfaces and integration components.
• Workwith application teams to define data contractsand integration requirements.
• Support future integrations and additional data consumers.
• Engineering Practices Follow modernsoftware engineering practices around: Git Code reviews Unit testing Integration testing CI/CD Logging Monitoring Documentation Develop reusable and maintainable data engineering components.
• Participate in technical designdiscussions and architecture reviews.
• Mentor other Data Engineers and contribute to engineering standards.
• Required Skills& Experience 6–8 years ofexperience in Data Engineering.
• Strong hands-onexperience with Apache Flink — mandatory/core requirement .
• Strong hands-on experience with Apache Kafka .
• Hands-on experience with Debeziumand CDC .
• Strong programming experience in Java or Scala .
• Strong SQL skills.
• Hands-on experience with Apache Airflowor another data orchestration framework .
• Strong understanding of distributed systemsand real-time data processing.
• Experience developing and supporting production-grade streaming pipelines.
• Strong understanding of Kafka: Topics Partitions Offsets Consumer groups Replication Retention Experience with data transformation, enrichment, filtering, aggregation, and event processing.
• Experience troubleshooting performance and reliability issues.
• Familiarity with cloud platforms and containerized environments.

Required qualifications

• 6–8 years

Preferred qualifications and skills

• Python experience is an advantage.
• Experience with analytical databases; ClickHouse is highly preferred .
• Advanced experience with Apache Flink , including: State management Checkpoints Savepoints Watermarks Event time Windows Backpressure State backends Experience with Apache Airflow, Dagster,Prefect, or Apache NiFi .
• Experience with Kafka Schema Registry.
• Experience with Avro, Protobuf, or JSON .
• Experience with Kubernetes.
• Experience with AWS / Azure / GCP.
• Experience with CI/CD pipelines.
• Experience with Docker and containerized applications.
• Experience with Terraform or other Infrastructure as Code tools.
• Experience with data observability and monitoring tools.
• Experience building high-volume, low-latency real-time data platforms .
• Experience with REST APIs and systemintegrations.

Employment details

• Location: Bengaluru / Hybrid Employment: Full-time Role Summary We are looking for a Lead Data Engineer with strong hands-on experience in real-time data streaming, event processing, CDC, data integration, and modern data engineering.
• The Lead Data Engineer will be responsible for designing, developing, and maintaining highperformance data pipelines using Apache Flink, Apache Kafka, Debezium, CDC, ClickHouse, and Apache Airflow.
• This is a hands-on engineering role requiring strong practical experience in building productiongrade streaming applications, developing data pipelines, troubleshooting distributed systems, and optimizing data processing workloads.
• The ideal candidate should be comfortable working across the full data pipeline—from source systems and CDC ingestion through Kafka and Flink processing to analytical storage, APIs, dashboards, and downstream integrations.
• Location: Bengaluru [KA] [India]

Additional role details

• Role: Lead Data Engineer