Senior Data Engineer

Role overview

Senior Data Engineer at Soti

Design, build, and maintain production grade ETL/ELT pipelines using SQL, Python, PySpark, and Spark SQL.

Responsibilities

• Own end to end data pipelines from ingestion through curated, analytics ready datasets.
• Maintain and enhance existing integrations to ensure stability, performance, and scalability.
• Troubleshoot complex pipeline failures and act as a key escalation point for production support.
• Collaborate with Lead Engineers and Data Architects on on-prem to cloud migration initiatives.
• Assess legacy pipelines and integrations, contributing to migration sequencing, refactoring, and risk mitigation.
• Ensure continuity of business-critical data flows during phased migrations and platform transitions.
• Build and operate solutions across hybrid data platforms, including SQL Server, Azure Synapse, and Microsoft Fabric.
• Develop and maintain Fabric Lakehouses, Warehouses, Pipelines, and Notebooks aligned with architectural standards.
• Optimize workloads for performance, reliability, and cost efficiency.
• Implement data validation, reconciliation, monitoring, logging, and alerting to ensure trusted data products.
• Apply disciplined engineering practices: version control, CI/CD, testing, and documentation.
• Improve the operational maturity and production readiness of data solutions.
• Mentor junior data engineers through code reviews, guidance, and best practice sharing.
• Promote consistent engineering standards and support team onboarding.
• Collaborate closely with BI teams, data owners, and governance stakeholders to ensure alignment with analytics needs.
• Identify and drive improvements in pipeline efficiency, migration effectiveness, and developer productivity.

Required qualifications

• 7+ years of experience in Data Engineering, Analytics Engineering, or a related technical role.
• Strong hands‑on experience with SQL, Python, PySpark, and Spark SQL in distributed data environments.
• Proven experience building and supporting enterprise‑scale ETL/ELT pipelines in the cloud.
• Strong experience with Azure‑based data platforms, including Azure Synapse Analytics and/or Microsoft Fabric.
• Solid understanding of cloud‑native data architectures, analytics engineering patterns, and medallion architectures.
• Experience with Git‑based source control, CI/CD pipelines, and automated testing for data platforms.
• Strong problem‑solving skills and experience supporting mission‑critical production data systems.
• Proven ability to collaborate across engineering and business teams and mentor junior engineers.
• Hands‑on experience with Microsoft Fabric Lakehouse and Spark workloads
• Experience with Azure Data Factory, Fabric Pipelines, or orchestration tools
• Familiarity with data governance, lineage, and certification in cloud platforms
• Relevant certifications such as: Microsoft Certified: Azure Data Engineer Associate, Microsoft Fabric Analytics Engineer
• SOTI does not charge any fees at any stage of the recruitment process.
• You can verify the authenticity of any SOTI job opportunities by visiting SOTI Careers .
• SOTI shall not be liable for any fraudulent recruitment activities carried out by unauthorized individuals or organizations.
• #LI-BK1
• If you want to bring your ideas to life, apply at SOTI today.
• Please note that SOTI does not accept unsolicited resumes from recruiters or employment agencies.
• In the absence of a signed Services Agreement with agency/recruiter, SOTI will not consider or agree to payment of any referral compensation or recruiter fee.

Preferred qualifications and skills

• Preferred / Nice to Have

Employment details

• Location: Kochi [India]

About the employer

• SOTI is committed to providing its employees with endless possibilities; learning new things, working with the latest technologies and making a difference in the world.