SENIOR DATA ENGINEER - Databricks

Role overview

SENIOR DATA ENGINEER - Databricks at Happiest Minds

Hands-on IC role focused on building reliable, performant data pipelines on Azure Databricks, implementing Delta Lake patterns, and contributing to production-grade data products for analytics.

Core Responsibilities ?

Build/maintain batch & basic streaming pipelines using PySpark/Spark SQL and Delta Lake following Medallion principles. ?

Ingest via Autoloader, ADF, REST, and file sources (e.g., SFTP/SharePoint). ?

Apply schema enforcement/evolution; add basic data quality checks and validations. ?

Use Unity Catalog for catalog/schema/table permissions; handle basic secrets via Key Vault/secret scopes. ?

Author clear code & docs; participate in PRs; contribute utilities; triage job issues and first-line performance fixes. ?

Operate simple Databricks Workflows (or ADF) and Git-based CI/CD basics.

Must-have Skills (Screening Focus) ?

Strong hands-on PySpark, Spark SQL, Delta Lake fundamentals; solid SQL. ?

Understanding of partitions, shuffles, joins, caching; Medallion basics. ?

Unity Catalog basics (catalog/schema/table permissions). ?

Autoloader basics; ADF familiarity or Workflows basics. ?

Git workflows; basic tests and logging; runbook hygiene.

Nice-to-have (Prefer, not Mandatory) ?

Exposure to DLT and Workflows orchestration patterns. ?

Photon awareness; cost-aware design exposure. ?

Healthcare/PII familiarity; Power BI integrations. 90-day Outcomes (Examples) ?

Ship 2 Silver?Gold pipelines with <1% failure rate; produce runbook. ?

Add basic data expectations (null, duplicate, type checks) with alerting on a critical table

Employment details

• Location: Bengaluru

Additional role details

• C3 ?
• Senior Data Engineer (6?8 years)