Associate Principal - Data Engineering

Experience 12 Years Role Overview We are seeking an experienced Databricks Data Architect to lead the design architecture and implementation of enterprisescale data platforms on Databricks The ideal candidate should possess strong expertise in modern data engineering practices dimensional data modeling and Databricks Lakehouse architecture The candidate will act as the technical owner of the data platform providing architectural guidance establishing best practices mentoring development teams and ensuring successful delivery of scalable highperformance data solutions Key Responsibilities Architecture Solution Design Design and implement enterprisescale data platforms using the Databricks Lakehouse architecture Define endtoend data architecture including ingestion transformation storage governance and consumption layers Establish architectural standards framework guidelines coding best practices and reusable design patterns Lead technical decisions related to scalability performance optimization reliability and security Conduct architecture reviews and provide recommendations to project teams Data Engineering Databricks Implementation Design and build scalable ETLELT pipelines using PySpark on Databricks Architect and implement Bronze Silver Gold data transformation frameworks Design and manage Delta Lake based storage architecture including partitioning optimization and data lifecycle management Implement Databricks Lakeflow pipelines for data orchestration and automation Define data quality validation reconciliation and monitoring frameworks Oversee CICD and DevOps practices for data engineering projects Data Modeling Analytics Enablement Design and implement enterprise data models using Dimensional Modeling techniques Develop Star Schemas Fact Tables Dimension Tables and conformed dimensions 
 Define business metrics KPI layers and semantic models 
 Architect and implement Metric Views and curated Gold datasets for reporting and analytics 
 Work closely with business stakeholders to translate requirements into scalable analytical models 
 Technical Leadership 
 Provide technical leadership and mentorship to data engineers and developers 
 Perform code reviews and ensure adherence to architecture standards 
 Troubleshoot complex technical issues and guide teams in solution design 
 Support estimation solution proposals and technical governance activities 
 Drive adoption of modern data engineering and cloudnative best practices 
 Required Technical Skills 
 Core Databricks Skills 
 Databricks Lakehouse Architecture 
 Databricks Workflows Lakeflow 
 Delta Lake 
 Unity Catalog 
 Databricks SQL 
 Performance Tuning Optimization 
 Data Engineering 
 PySpark Spark SQL ETL ELT Design Data Pipeline Development Data Quality Frameworks Batch and Incremental Processing Data Modeling Dimensional Data Modeling Star Schema Design Fact Dimension Modeling Slowly Changing Dimensions SCD Metric View Design Analytical Data Modeling Cloud Integration Azure Databricks preferred ADLS Gen2 Azure Data Factory ADF Eventbased and Batch Ingestion Patterns Data Governance and Security Programming Python SQL Git CICD Frameworks Desired Skills Data Warehouse Modernization experience Experience migrating legacy EDW solutions to Databricks Exposure to Data Mesh and Medallion Architecture Knowledge of Data Governance Metadata Management and Data Cataloging Experience supporting BI platforms such as Power BI Tableau or Looker Leadership Expectations The candidate should be capable of Owning the complete Databricks implementation lifecycle Defining targetstate architecture and roadmap Reviewing designs and code produced by team members Guiding developers on PySpark optimization and data modeling best practices Driving technical discussions with client architects and business stakeholders Ensuring scalability maintainability and longterm sustainability of the platform Providing architectural governance across multiple workstreams