Lead Data Engineer

Role overview

Lead Data Engineer at Tudip Technologies

Lead the design, development, and maintenance of robust and scalable data pipelines using Databricks.

The ideal candidate will lead the data engineering team, collaborate with cross-functional stakeholders, and ensure high-quality, reliable, and secure data solutions that support business and analytics needs.

Responsibilities

• Lead Data Engineer
• Architect end-to-end data solutions including data ingestion, transformation, storage, and analytics.
• Build and optimize data workflows using Databricks, Spark, and cloud data platforms.
• Guide and mentor a team of data engineers, ensuring best practices and code quality.
• Collaborate with product managers, data scientists, analysts, and business teams to understand data requirements.
• Ensure data quality, governance, security, and compliance standards are met.
• Optimize performance and cost of data pipelines and platforms.
• Review technical designs, code, and deployments.
• Troubleshoot and resolve complex data-related issues.
• Participate in strategic planning and roadmap discussions for data platforms.

Required qualifications

• Hands-on experience with Databricks is mandatory for this role.
• Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field. 7+ years of experience in data engineering or related roles.
• Mandatory hands-on experience with Databricks (Delta Lake, Databricks SQL, notebooks, jobs, workflows).
• Strong experience with Apache Spark and distributed data processing.
• Proficiency in SQL and programming languages such as Python, Scala, or Java.
• Experience with cloud platforms (AWS, Azure, or GCP).
• Expertise in data warehousing solutions (Snowflake, Redshift, Big Query, etc.).
• Experience with ETL/ELT tools and orchestration frameworks (Airflow, dbt, etc.).
• Strong understanding of CI/CD, version control, and Dev Ops practices.
• Excellent leadership, communication, and problem-solving skills.

Preferred qualifications and skills

• Experience leading or mentoring data engineering teams.
• Knowledge of real-time data processing and streaming (Kafka, Spark Streaming, etc.).
• Exposure to BI and analytics tools.
• Experience working in Agile/Scrum environments.

Employment details

• Location: India

Additional role details

• We are looking for an experienced Lead Data Engineer to design, build, and manage scalable data pipelines and platforms.