Specialist - Data Engineering

Key Responsibilities Design and implement complex data pipelines using Dataflow Apache Beam and Dataproc Spark Lead Cloud Composer Airflow orchestration design optimizing DAG structure dependencies and retry strategies Drive advanced Python and SQL implementations including reusable frameworks and utility components Build and standardize custom Dataflow Beam templates and reusable pipeline patterns Perform deep performance tuning for BigQuery query optimization partitioning strategies cost optimization Troubleshoot complex pipeline failures and performance bottlenecks across processing and orchestration layers Review code and design artifacts produced by engineers provide technical guidance and mentoring Own technical quality for assigned modules including nonfunctional aspects performance reliability scalability Act as the first point of escalation for technical issues before involving architects Collaborate with architects to understand design intent and ensure correct implementation Mandatory Technical Skills Advanced GCP Knowledge Strong understanding of IAM strategy service isolation and projectlevel design considerations Programming Querying Advanced proficiency in Python and complex GoogleSQL Orchestration Strong handson expertise with Cloud Composer Airflow including best practices and optimization Data Processing Deep handson experience with Dataflow and Apache Beam Strong working knowledge of Dataproc Spark internals Analytics Platform Advanced experience with BigQuery including performance and cost optimization techniques Good to Have Exposure to streaming data pipelines and PubSubbased ingestion Familiarity with CICD Gitbased version control and basic DevOps practices Understanding of data quality checks logging and monitoring patterns Exposure to AgileScrum delivery models GCP certifications such as Associate Cloud Engineer or Professional Data Engineer nice to have