Senior Data Engineer

Role overview

Senior Data Engineer at Quantiphi

Databricks & Lakehouse Management: Manage Databricks workflows, clusters, and Delta Lake.

Responsibilities

• ETL/ELT Data Pipeline Engineering: Design, build, and optimize scalable real-time and batch data pipelines using Python, SQL, PySpark, Spark Structured Streaming, Delta Live Tables, and Medallion Architecture (Bronze, Silver, Gold layers) for high-performance analytics.Must have hands-on experience working with terabyte-scale (TB) datasets and designing distributed data processing solutions.
• Implement Unity Catalog for metadata management, fine-grained access control, OAuth-based authentication, and enterprise security compliance.
• Cloud Infrastructure & Migration: Experience working with at least one major cloud platform (GCP, AWS, or Azure).
• Lead cloud migrations from legacy databases (Oracle, Teradata, SQL Server) to cloud environments (GCS, AWS S3, Azure Data Lake Storage, Snowflake, Redshift).
• CI/CD & DevOps Automation: Build end-to-end CI/CD pipelines using Terraform and GitHub Actions to automate data pipeline deployments, infrastructure provisioning, and proactive monitoring.
• Web Services & Application Scripting: Develop and deploy custom data processing scripts and data-driven APIs using Python and web frameworks like Flask.
• Containerization & Innovation: Utilize Docker for containerized processing and evaluate emerging technologies for data-intensive machine learning tasks.
• Stakeholder Collaboration: Work closely with business stakeholders and cross-functional engineering teams to translate business requirements into technical architectures and data-driven insights.
• Skills expectations:
• Core Data Engineering & Big Data: Advanced hands-on experience in Python, SQL, PySpark, and expert-level SQL query optimization for massive datasets.
• Databricks Expertise: Strong proficiency in Databricks functionalities, including Workflows, Unity Catalog, Delta Lake, and Delta Live Tables.
• Cloud Platform Services: Deep familiarity across major cloud environments (GCP, AWS S3/Redshift, Azure ADLS Gen2).
• DevOps & Infrastructure: Solid experience in Terraform, GitHub Actions, Docker, and CI/CD best practices for data platforms.
• Web Frameworks: Practical experience with Python web frameworks such as Flask.
• Collaborative & Agile Mindset: Strong problem-solving, communication, and ownership skills to guide projects and mentor team members in a fast-paced environment.
• If you like wild growth and working with happy, enthusiastic over-achievers, you'll enjoy your career with us !

Employment details

• Experience Level: 4-7 Years
• Work
• Shift: 6:00 PM - 3:00 AM IST (US Business Hours Shift)
• location: Remote
• Notice Period: 0-15 days(Looking for immediate joiners or currently serving Notice period)
• Role &
• Location: IN KA Bengaluru, IN MH Mumbai Quantum, IN KL Trivandrum, IN MH Mumbai ISD, IN MH Mumbai Eureka

Additional role details

• While technology is the heart of our business, a global and diverse culture is the heart of our success.
• We love our people and we take pride in catering them to a culture built on transparency, diversity, integrity, learning and growth.
• If working in an environment that encourages you to innovate and excel, not just in professional but personal life, interests you- you would enjoy your career with Quantiphi!
• Role: Senior Data Engineer - Python, Spark & Databricks