Big Data Engineer – Java, Apache Spark, AWS EMR, Lambda, EKS & Airflow

Job Summary Synechron is seeking a Big Data Engineer with 5+ years of experience in designing, developing and deploying scalable and reliable data-processing solutions on AWS using Java and Apache Spark. The role will focus on implementing big data processing pipelines using Apache Spark on AWS EMR, developing and deploying serverless applications using AWS Lambda, utilizing Amazon EKS for container orchestration and microservices management, and designing workflow orchestration using Apache Airflow. The position will contribute to business objectives by delivering reliable, scalable and maintainable data-processing solutions that support efficient data operations and application services. Software Requirements Required Java: Strong hands-on experience in Java development. Apache Spark: Proficiency in Apache Spark for distributed data processing. AWS EMR: Experience implementing big data processing pipelines using Apache Spark on AWS EMR. AWS Lambda: Experience developing and deploying serverless applications using AWS Lambda. AWS Service Integration: Experience integrating AWS Lambda with other AWS services. Amazon EKS: Experience utilizing Amazon EKS for container orchestration and microservices management. Apache Airflow: Experience designing and implementing workflow orchestration for complex data pipelines. AWS: Experience designing, developing and deploying scalable and reliable data-processing solutions on AWS using Java and Spark. Project-Supported Versions: Ability to work with the versions of Java, Apache Spark, AWS services and related tools supported by the project. Preferred No additional preferred software skills are specified in the requirements. Additional experience relevant to Java, Apache Spark, AWS, AWS EMR, AWS Lambda, Amazon EKS or Apache Airflow may be considered beneficial. Overall Responsibilities Design, develop and deploy scalable and reliable data-processing solutions on AWS using Java and Apache Spark. Implement and maintain big data processing pipelines using Apache Spark on AWS EMR. Develop and deploy serverless applications using AWS Lambda and integrate them with other AWS services. Utilize Amazon EKS for container orchestration and microservices management. Design, implement and maintain Apache Airflow workflows for complex data pipelines. Develop maintainable Java applications and data-processing components aligned with technical requirements. Contribute to solution design, technical discussions and implementation decisions within agreed project standards. Test, troubleshoot and resolve application, pipeline, integration and deployment issues. Support reliable operation of data-processing solutions across relevant development, testing and production environments. Monitor application and pipeline behavior and contribute to improvements in scalability, reliability and maintainability. Collaborate with engineering, architecture, QA and delivery teams to clarify requirements and coordinate implementation activities. Contribute to code reviews, technical documentation, release activities and knowledge sharing. Apply appropriate data protection and cloud security practices during development and deployment. Use AWS resources efficiently and consider sustainable practices that reduce unnecessary compute, storage and network consumption. Deliver assigned work in accordance with agreed priorities, timelines and quality expectations. Technical Skills (By Category) Programming Languages Essential Strong hands-on experience in Java development. Ability to develop clean, modular, reusable and maintainable Java code. Experience using Java to develop data-processing solutions and supporting application components. Ability to debug Java code and resolve functional, integration and deployment issues. Preferred Additional experience with languages used in data engineering may be considered beneficial. Databases and Data Management Essential Experience implementing big data processing pipelines. Understanding of distributed data processing using Apache Spark. Ability to design data-processing workflows that support reliability and scalability. Ability to manage pipeline execution, dependencies and processing failures. Preferred Additional experience with data storage, transformation, validation or integration may be considered beneficial. Cloud Technologies Essential Experience designing, developing and deploying data-processing solutions on AWS. Experience with AWS EMR for Apache Spark-based big data processing. Experience with AWS Lambda for serverless application development and deployment. Experience integrating AWS Lambda with other AWS services. Experience with Amazon EKS for container orchestration and microservices management. Experience using project-supported AWS service configurations and versions. Preferred Additional experience relevant to AWS-based data-processing and application deployment may be considered beneficial. Frameworks and Libraries Essential Apache Spark: Proficiency in distributed data processing. Apache Airflow: Experience designing and implementing workflow orchestration for complex data pipelines. Experience integrating Apache Spark with AWS EMR. Experience using frameworks and libraries compatible with project-supported Java, Spark and AWS versions. Preferred Additional experience developing reusable data-processing components may be considered beneficial. Development Tools and Methodologies Essential Experience with the design, development and deployment lifecycle for data-processing solutions. Experience deploying and maintaining applications and pipelines in AWS environments. Ability to troubleshoot data pipelines, serverless applications, containerized services and workflow processes. Ability to document technical solutions, workflows and deployment-related activities. Ability to collaborate with relevant technical and delivery stakeholders. Preferred No additional development tools or methodologies are specified in the requirements. Experience with automated testing, deployment or monitoring practices may be considered beneficial. Security Protocols Essential Understanding of secure development practices for cloud-based applications and data-processing solutions. Awareness of appropriate access control for AWS services and deployed applications. Ability to protect data during processing, transmission and storage. Ability to manage application configuration and credentials securely. Ability to apply secure logging and error-handling practices. Preferred No specific additional security protocols are specified in the requirements. Experience with cloud security controls for serverless, containerized and distributed data-processing solutions may be considered beneficial. Experience Requirements 5+ years of experience in big data engineering, Java development, cloud engineering or a related technical role. Experience designing, developing and deploying scalable and reliable data-processing solutions on AWS using Java and Apache Spark. Experience implementing big data processing pipelines using Apache Spark on AWS EMR. Experience developing and deploying serverless applications using AWS Lambda and integrating them with other AWS services. Experience utilizing Amazon EKS for container orchestration and microservices management. Experience designing and implementing workflow orchestration using Apache Airflow for complex data pipelines. Strong hands-on experience in Java development and proficiency in Apache Spark for distributed data processing. Experience with AWS services including EMR, Lambda, EKS and Airflow. Experience supporting the deployment, troubleshooting and maintenance of AWS-based data-processing solutions is relevant to the role. Candidates may qualify through equivalent experience in Java-based data engineering, distributed data processing, AWS engineering or cloud-based application development that demonstrates the required capabilities. Day-to-Day Activities Develop, test and deploy Java and Apache Spark data-processing pipelines on AWS EMR and troubleshoot pipeline issues. Build and maintain AWS Lambda applications, Amazon EKS services and Apache Airflow workflows in collaboration with technical teams. Participate in technical discussions, requirement reviews, code reviews, planning meetings, defect resolution and release activities. Make implementation decisions within agreed technical and security standards, document deliverables and communicate progress, risks and dependencies. Qualifications A bachelor’s degree in Computer Science, Information Technology, Engineering or a related field is preferred; equivalent relevant experience may be considered. Certifications in AWS, Java, Apache Spark, data engineering or related technical disciplines are preferred but not mandatory. Practical training or demonstrated experience in Java, Apache Spark, AWS EMR, AWS Lambda, Amazon EKS and Apache Airflow is required. Training or experience in distributed systems, serverless applications, container orchestration and cloud-based data processing is beneficial. Commitment to continuous professional development in Java, Apache Spark, AWS services, Apache Airflow and related data engineering practices is expected. Professional Competencies Applies critical thinking to understand data-processing requirements and design scalable and reliable technical solutions. Uses structured problem-solving to troubleshoot pipeline failures, application defects, integration issues and deployment problems. Collaborates effectively with engineering, architecture, QA and delivery teams and contributes to shared technical outcomes. Communicates implementation decisions, progress, risks, dependencies and technical concerns clearly to relevant stakeholders. Adapts to evolving Java, Apache Spark, AWS, serverless, container orchestration and workflow-orchestration technologies. Manages priorities, technical dependencies and delivery timelines while maintaining solution quality, reliability and maintainability. SYNECHRON’SDIVERSITY & INCLUSION STATEMENT Diversity & Inclusion are fundamental to our culture, and Synechron is proud to be an equal opportunity workplace and is an affirmative action employer. Our Diversity, Equity, and Inclusion (DEI) initiative ‘Same Difference’ is committed to fostering an inclusive culture – promoting equality, diversity and an environment that is respectful to all. We strongly believe that a diverse workforce helps build stronger, successful businesses as a global company. We encourage applicants from across diverse backgrounds, race, ethnicities, religion, age, marital status, gender, sexual orientations, or disabilities to apply. We empower our global workforce by offering flexible workplace arrangements, mentoring, internal mobility, learning and development programs, and more. All employment decisions at Synechron are based on business needs, job requirements and individual qualifications, without regard to the applicant’s gender, gender identity, sexual orientation, race, ethnicity, disabled or veteran status, or any other characteristic protected by law. Candidate Application Notice At Synechron, we believe in the power of digital to transform businesses for the better. Our global consulting firm combines creativity and innovative technology to deliver industry-leading digital solutions. Synechron’s progressive technologies and optimization strategies span end-to-end Artificial Intelligence, Consulting, Digital, Cloud & Dev Ops, Data, and Software Engineering, servicing an array of noteworthy financial services and technology firms. Through research and development initiatives in our Fin Labs we develop solutions for modernization, from Artificial Intelligence and Blockchain to Data Science models, Digital Underwriting, mobile-first applications and more. Over the last 20+ years, our company has been honored with multiple employer awards, recognizing our commitment to our talented teams. With top clients to boast about, Synechron has a global workforce of 14,500+, and has 58 offices in 21 countries within key global markets. For more information on the company, please visit our website or Linked In community. At Synechron, we are committed to integrating sustainability into our business strategy, ensuring responsible growth while minimizing environmental impact. Employees play a key role in driving our sustainability initiatives, from reducing our carbon footprint to fostering ethical and sustainable business practices across global operations. All positions are required to adhere to our Sustainability and Health Safety standards, demonstrating a commitment to environmental stewardship, workplace safety, and sustainable practices. Not finding the right fit? Let us know you're interested in a future opportunity by clicking Get Started below or create an account by clicking 'Sign In' at the top of the page to set up email alerts as new job postings become available that meet your interest!