Senior Data Engineering

Role overview

Senior Data Engineering at Mphasis

for Data Engineer – Databricks Data Platform

Responsibilities

• Build and maintain data pipelines using Databricks, Spark, and Delta Lake on AWS.
• Ingest data from databases, APIs, event streams, and flat files.
• Design and implement Medallion Architecture (Bronze / Silver / Gold layers).
• � Write data transformation logic in PySpark, Spark SQL, and Python.
• � Implement data quality checks, validation rules, and reconciliation processes.
• Optimise Spark workloads for performance, cost, and scalability.
• Integrate AWS services (S3, Glue, Lambda, Redshift, IAM) into data solutions.
• Support CI/CD for pipelines using Git, DevOps tooling, and release processes.
• Monitor pipelines, triage failures, and restore services within agreed SLAs.
• Maintain technical documentation: data models, pipeline flows, runbooks.
• Adhere to data security, privacy, and regulatory compliance standards.
• Must-Have Skills
• � Databricks: Workflows, Delta Live Tables, Unity Catalog, Databricks SQL.
• Apache Spark: PySpark and Spark SQL for large-scale data processing.
• Delta Lake: ACID transactions, schema evolution, time travel, merge.
• Medallion Architecture or equivalent layered data platform design.
• Python for pipeline development and transformation logic.
• � AWS: S3, Glue, IAM, EC2, Lambda.
• � Batch, micro-batch, and streaming ingestion (Structured Streaming / Kafka).
• Data modelling: star schema, snowflake schema, Data Vault.
• CI/CD, Git, version control, and Agile delivery practices.
• Good-to-Have Skills
• Unity Catalog for data governance and lineage.
• AWS Kinesis for real-time streaming.
• Data Warehouse and Informatica PWC Knowledge
• dbt for SQL-based transformation workflows.
• � AWS Redshift or Athena for cloud data warehousing.
• � Terraform or AWS CloudFormation for infrastructure-as-code.
• � AWS Step Functions for workflow orchestration.
• Data observability tools (Monte Carlo, AWS Glue Catalog, Atlas).
• Industry experience in insurance or financial services.
• Databricks or AWS Data Analytics certification.
• Technical Environment
• Platform: Databricks Lakehouse, Delta Lake, Unity Catalog.
• Languages: Python, PySpark, SQL, Spark SQL.
• � Ingestion: Auto Loader, APIs, JDBC, CDC tools.
• � Processing: Apache Spark, Structured Streaming, Delta Live Tables.
• Cloud: AWS.
• Orchestration: Databricks Workflows, AWS Step Functions.
• DevOps: Git, CI/CD, Terraform, GitHub Actions.
• Analytics: Power BI, Tableau, Looker.
• Experience and Qualifications
• Bachelor's in Computer Science, Engineering, IT, or equivalent.
• 4–8 years in data engineering, ETL/ELT, and cloud data platforms.
• � Production-grade pipeline delivery on AWS.
• � Agile or DevOps-driven delivery environment.

Employment details

• Location: Pune [India]