Data Engineer-Data Platforms-Azure

Role overview

Data Engineer-Data Platforms-Azure at IBM

Responsibilities

• You’ll work with leading companies across industries, helping them shape their hybrid cloud and AI journeys.
• You’ll be encouraged to challenge the norm, explore new ideas, and create innovative solutions that deliver real results.
• Design and Build Data Pipelines: Design, build, and operate batch and real-time data pipelines using Azure services such as Azure Synapse Analytics, Azure Data Factory, Azure DataBricks, and Event Hub.

Required qualifications

• Only Development Experience required NO support Experience. 2-3 Implementation experience is required.

Employment details

• India Data & Analytics Hybrid Professional Bangalore, IN Bangalore (0063) IBM India Private Limited Analytics Data Integration Data Migration Data Modeling Data Processing Data Validation Design Dimensional Modeling Extract, Transform, Load Modeling Optimization Pipelines Python Structured Query Language Scala Spark Storage Unity Azure Data Mapping Stream Processing Data Engineering Data ingestion Pyspark Pytest Data Cleansing Data pipeline Apache Airflow Azure Cloud Azure Data Factory Data Platform Databricks Open Source Streaming Data Azure Cosmos DB Azure Synapse Analytics Data Orchestration Real-time Data Azure Functions Snowflake Schema
• Location: Bangalore [India]

Application details

• Utilize Open Source Technologies: Apply proficiency in open source technologies like Apache Airflow and dbt, Spark / Python, or Spark / Scala to support data engineering solutions.

About the employer

• With support from our strategic partners, robust IBM technology, and Red Hat, you’ll have the tools to drive meaningful change and accelerate client impact.

Additional role details

• A career in IBM Consulting is built on long-term client relationships and close collaboration worldwide.
• At IBM Consulting, curiosity fuels success.
• Our culture of growth and empathy focuses on your long-term career development while valuing your unique skills and experiences.
• As a Data Engineer specializing in Data Platforms on Azure, you will advise on, develop, and maintain data engineering solutions on the Azure Cloud ecosystem.
• You will design, build, and operate batch and real-time data pipelines using various Azure services.
• Your primary responsibilities will include:
• Develop Data Layer: Design, build, and operate the data layer on Azure Synapse Analytics, SQL DW, and Cosmos DB.
• Implement Azure Data Platform: Implement and maintain Azure Data Platform components, including ADLS2, Blob Storage, SQLDW, Synapse Analytics with Spark and SQL, Azure functions with Python, Azure Purview, and Cosmos DB.
• Ensure Seamless Operations: Operate and maintain data engineering solutions to ensure seamless batch and real-time data processing.
• Minimum 6 years in Data Engineering (Azure-focused).
• Exposure to Azure Toolset: Proficiency in Azure services such as Azure Synapse Analytics, Azure Data Factory, Azure DataBricks, and Event Hub to design, build, and operate batch and real-time data pipelines.
• Data Engineering on Azure: Experience working with Azure Data Platform components, including ADLS2, Blob Storage, SQLDW, Synapse Analytics with Spark and SQL, Azure functions with Python, Azure Purview, and Cosmos DB.
• Open-Source Technologies: Exposure to open-source technologies like Apache Airflow and dbt, PySpark / Python, or Spark to support data engineering solutions.Data Pipeline Development: Experience developing and maintaining data engineering solutions on the Azure Cloud ecosystem, including designing, building, and operating batch and real-time data pipelines.
• Azure Data Layer Design: Experience designing, building, and operating the data layer on Azure Synapse Analytics, SQL DW, and Cosmos DB Databricks Expertise: Hands-on experience with PySpark, Delta Live Tables, Unity Catalog, and performance optimization.
• SQL DB, Databricks, Pytest Analytics.
• Data Ingestion: Batch ingestion, Real-time ingestion, real-time processing, Streaming data pipelines, Event-driven ingestion, Incremental load, Full load, Delta load Data Processing Capabilities: Batch, real-time, and stream processing, Incremental, micro-batch, and event-driven processing Data Manipulation & Processing: Data cleansing, transformation, standardization, Data validation, quality checks, deduplication, reconciliation, Data mapping, merging, splitting, masking Data Integration: Strong hands-on in ETL & ELT pipelines, Data migration and transformation across systems, Data orchestration Data Modelling: Dimensional modelling, Star & Snowflake schema, CD Type 1 & Type 2,ER modelling (Entity Relationship Modelling)
• Proficiency in Azure Services: Exposure to Azure services such as Azure Synapse Analytics, Azure Data Factory, Azure DataBricks, and Event Hub to design, build, and operate batch and real-time data pipelines.
• Open Source Technologies: Exposure to open source technologies like Apache Airflow and dbt, Spark / Python, or Spark / Scala to support data engineering solutions.