Senior Specialist - Data Engineering

Role: PySpark Application Support Lead Experience: 8-12 Years Location :Anywhere in India Focus Production Support Troubleshooting Optimization Team Leadership Role Summary Leadership Skills and proven experience in leading technical teams handling escalations and conducting knowledgesharing sessions The Pyspark Application support is responsible for the monitoring and troubleshooting Pyspark Jobs on the Cloudera environment This role ensures high availability performance and security of ETL services while supporting developers analysts and business users across environments Prod and Nonprod Technical Skills Primary PySpark Application support Handson experience with Pyspark jobs development ETL support Monitor PySpark batch jobs streaming pipelines and scheduled workflows Airflow ADF Oozie etcChecking for failures and performance degradation Troubleshoot job failures performance degradation schema mismatches data quality issues Resolve errors related to Spark SQL DataFrame transformations cluster configuration and storage layers Handle failures caused by upstreamdownstream systems Kafka Delta Lake Hive S3ADLS etc Coordinating with development team for fixing the failures Coordinating with QA team on data validation failures Supporting in production release and change management process Incident Problem Management Root cause analysis for recurring issues Performance tuning at a basic level cache queues agents Familiarity with ITIL processes for AMS support Tools Ticketing systems ServiceNowJira Monitoring tools Version control basics GitSVN for migration scripts Soft Skills Ability to work with client stakeholders Good communication with analytical and problemsolving skills Ability to Effectively interact with stakeholders colleagues and other members of the team 
 Demonstrated ability to be organized and selfdirected 
 Strong interpersonal skills with the ability to work cross functionally 
 Responsibilities 
 Monitoring Maintenance 
 Proactively monitor Informatica environments for uptime and performance 
 Execute scheduled health checks and housekeeping tasks 
 Maintain logs and escalate anomalies promptly 
 Incident Management 
 Respond to incidents within SLA timelines 
 Perform initial triage resolve L2 issues and escalate L3 as needed 
 Document RCA and preventive measures 
 User Support 
 Assist users with access issues report failures and portal navigation 
 Manage user provisioning and security roles 
 Change Release Support 
 Support deployments and migrations between environments 
 Validate postdeployment checks and rollback plans 
 Compliance Documentation 
 Ensure adherence to security and audit requirements Maintain SOPs runbooks and knowledge base for AMS operations Production Support Incident Management Act as the primary point of contact for highpriority incidents debugging and resolving complex PySpark application issues in production Technical Leadership Lead the efforts to design build and configure improvements to existing data pipelines and applications Team Management Mentorship Guide mentor and manage a team of support specialistsdevelopers to achieve operational excellence and meet project deadlines Performance Optimization Analyze existing PySpark applications and data pipelines for performance bottlenecks and implement optimization techniques eg Spark UI analysis tuning executors Stakeholder Interaction Collaborate with crossfunctional teams developers Data Engineers DevOps to ensure seamless application functionality Documentation Best Practices Create maintain and update technical documentation standard operating procedures SOPs and ensure code compliance with best practices Automation Implement monitoring improvements and automate recurring manual tasks