SENIOR DATA ENGINEER - Azure Databricks

Role overview

SENIOR DATA ENGINEER - Azure Databricks at Happiest Minds

We are seeking a highly skilled Senior Data Engineer with extensive experience in Azure Databricks, Azure Data Factory, and Palantir to join our dynamic team in Hyderabad.

The ideal candidate will have a strong background in data engineering, with a proven track record of designing and implementing data solutions that drive business insights and decision-making.

This role requires a deep understanding of data architecture, ETL processes, and cloud-based data platforms.

Responsibilities

• Design, develop, and maintain scalable data pipelines using Azure Databricks and Azure Data Factory.
• Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver high-quality data solutions.
• Implement data integration processes to ingest, transform, and store data from various sources into data lakes and warehouses.
• Optimize data workflows and ensure data quality, reliability, and performance.
• Utilize Palantir for data analysis and visualization, providing actionable insights to the business.
• Monitor and troubleshoot data pipeline issues, ensuring timely resolution and minimal downtime.
• Stay updated with the latest industry trends and technologies related to data engineering and cloud computing.
• Mentor junior data engineers and contribute to team knowledge sharing.
• Mandatory Skills: Proven experience with Azure Databricks (deal breaker skill).
• Strong proficiency in Azure Data Factory for data integration and orchestration.
• Experience with Palantir for data analysis and visualization.
• Solid understanding of data modelling, ETL processes, and data warehousing concepts.
• Proficient in programming languages such as Python, SQL, or Scala.
• Good understanding of Generative AI and Machine Learning concepts, including LLM, RAG, Vector Databases, Embeddings, Token Management, Feature Selection, Model Evaluation, and Neural Network fundamentals

Required qualifications

• 5-8 Years
• Bachelor's degree in Computer Science, Information Technology, or a related field. 5-8 years of experience in data engineering or a related role.
• Strong analytical and problem-solving skills.
• Excellent communication and collaboration abilities.
• If you are passionate about data engineering and want to work in a fast-paced environment, we encourage you to apply for this exciting opportunity.

Preferred qualifications and skills

• Ability to demonstrate practical knowledge of ML algorithms such as KNN, Linear Regression, CNN
• Experience with cloud platforms such as Azure, AWS, or Google Cloud.
• Familiarity with big data technologies like Apache Spark, Hadoop, or Kafka.
• Knowledge of data governance and data security best practices.
• Experience with CI/CD pipelines and DevOps practices in data engineering.

Employment details

• Location: Hyderabad

Additional role details

• Job Title: Senior Data Engineer Years of