Data Engineer-Data Platforms-AWS

Role overview

Data Engineer-Data Platforms-AWS at IBM

Responsibilities

• You’ll be encouraged to challenge the norm, explore new ideas, and create innovative solutions that deliver real results.
• Design and Develop Data Pipelines: Design, build, and operate batch and real-time data pipelines using AWS services such as AWS EMR, AWS Glue, Glue Catalog, and Kinesis, ensuring seamless integration and operation of data engineering solutions.
• Create Data Layers: Create data layers on AWS RedShift, Aurora, and DynamoDB, and migrate data using AWS DMS.
• Manage Data Services: Schedule and manage data services on the AWS Platform, ensuring efficient operation of data engineering solutions.
• Develop Batch and Real-time Pipelines: Develop batch and real-time data pipelines for Data Warehouse and Datalake, utilizing AWS Kinesis and Managed Streaming for Apache Kafka.

Employment details

• Location: PUNE [India]

About the employer

• A career in IBM Consulting is built on long-term client relationships and close collaboration worldwide.
• With support from our strategic partners, robust IBM technology, and Red Hat, you’ll have the tools to drive meaningful change and accelerate client impact.
• India Software Engineering Hybrid Professional PUNE, IN (0063) IBM India Private Limited

Additional role details

• You’ll work with leading companies across industries, helping them shape their hybrid cloud and AI journeys.
• At IBM Consulting, curiosity fuels success.
• Our culture of growth and empathy focuses on your long-term career development while valuing your unique skills and experiences.
• As a Data Engineer specializing in Data Platforms on AWS, you will advise on, develop, and maintain data engineering solutions on the AWS Cloud ecosystem.
• You will design, build, and operate batch and real-time data pipelines using various AWS services.
• Your primary responsibilities will include:
• Utilize Open Source Technologies: Utilize open source technologies like Apache Airflow and dbt, Spark / Python or Spark / Scala on AWS Platform to support data engineering solutions.
• Exposure to AWS Toolset: Experience working with AWS services such as AWS EMR, AWS Glue, Glue Catalog, and Kinesis to design, build, and operate batch and real-time data pipelines.
• Data Pipeline Development: Exposure to developing batch and real-time data pipelines for Data Warehouse and Datalake, utilizing AWS Kinesis and Managed Streaming for Apache Kafka.
• Data Layer Creation: Experience working with AWS RedShift, Aurora, and DynamoDB to create data layers and migrate data using AWS DMS.
• Open Source Technologies: Exposure to utilizing open source technologies like Apache Airflow and dbt, Spark / Python or Spark / Scala on AWS Platform to support data engineering solutions.
• Data Service Management: Experience scheduling and managing data services on the AWS Platform, ensuring efficient operation of data engineering solutions.
• Proficiency with AWS Databrew: Experience working with AWS Glue Databrew to support data engineering solutions, including data preparation and data quality tasks.
• Knowledge of Lambda Functions: Exposure to using Lambda functions with Python to support data engineering solutions, including data processing and data transformation tasks.
• Familiarity with RedShift Spectrum: Experience working with RedShift Spectrum to support data engineering solutions, including data warehousing and data analytics tasks.