Data Engineer

Data Engineer 4-6 years experience / Chennai We are looking for a skilled Data Engineer to design, build, and maintain scalable data pipelines and platforms. You will work closely with data scientists, analysts, and business stakeholders to ensure reliable, high-quality data across our ecosystem. Responsibilities · Design and develop robust ETL/ELT pipelines using PySpark and SQL for large-scale data processing. · Build, manage, and optimise data workflows on Databricks, including notebooks, jobs, and cluster configurations. · Collaborate with analytics and ML teams to deliver clean, well-documented datasets. · Monitor pipeline performance, troubleshoot failures, and ensure data quality and integrity. · Define and implement data governance and best practices across the data platform. · Participate in design reviews and contribute to architectural decisions. Requirements · 4-6 years of hands-on experience in data engineering or a related role. · Strong proficiency in PySpark for large-scale distributed data processing. · Advanced SQL skills - query optimisation, window functions, CTEs, and complex joins. · Solid experience with Databricks - Delta Lake, workflows, Unity Catalog preferred. · Experience working with cloud platforms (AWS/Azure). Core skills PySpark/SQL/Databricks/AWS / Azure /Python/Git Nice to have · Experience with streaming frameworks such as Apache Kafka or Spark Streaming. · Exposure to ML pipelines and feature engineering workflows. Requirements · 4-6 years of hands-on experience in data engineering or a related role. · Strong proficiency in PySpark for large-scale distributed data processing. · Advanced SQL skills - query optimisation, window functions, CTEs, and complex joins. · Solid experience with Databricks - Delta Lake, workflows, Unity Catalog preferred. · Experience working with cloud platforms (AWS/Azure).