Browse current jobs

Data Engineer - Datastore Migration

Bounteous · Bangalore

Apply on employer website

Source and collection record

Read the employer source

Last observed by our collector: . This is not a manual review or a confirmation from the employer.

Employer requisition: f84c5529-2b7e-4e5d-b9bc-c0c907e9eab1

Job type and experience labels are automated discovery categories. The employer determines eligibility. Original experience wording was not retained separately; check the current employer page.

How sourcing and classification work · Report a discrepancy

Your application evidence checklist

Mark the statements you can support after reading the employer source. This checklist stays on this page; it does not score your eligibility or submit an application.

0 of 2 checks marked. Keep unresolved requirements as questions for the employer.

Collected role description

Role Overview The Engineer will be part of the datastore-migration Factory team, responsible for the end-to-end datastore migration from an on-prem DataLake to an AWS-hosted LakeHouse. This is a high-visibility and crucial project. Basic Qualifications Bachelor's or Master's in Computer Science, Applied Mathematics, Engineering, or a related quantitative field. Technical Skills A total of 8+ years of experience in the field of Data Engineering. Minimum of 3-5 years of professional "hands-on-keyboard" coding experience in a collaborative, team-based environment. In-depth proficiency in SQL, Python/Java, and working experience using RESTful APIs with Swagger and automation; understand request/response and investigate using developer tools. Good knowledge of the on-prem Data Lake built on the Hadoop ecosystem with HDFS, and understanding of how the ecosystem operates. Experience with AWS S3 data staging and synchronous/asynchronous data sync techniques into a Lakehouse. Excellent experience handling CI/CD pipelines, stages, child pipeline management, investigating pipeline logs, and root cause analysis. Ability to execute end-to-end migration of Legacy Data Lake data stores to a Lakehouse in the cloud using available tooling. Ability to investigate issues with tooling during data migration, perform root cause analysis, and report issues promptly to different collaborating tooling teams, tracking them to closure. Experience translating and optimizing legacy SQL and Spark-based consumption patterns (raw and modeled) for compatibility with Snowflake and Iceberg. Strong understanding of milestoning and temporal data modeling (unitemporal, bitemporal, etc.); Temporal Data Modeling - managing state changes over time (e.g., SCD Type 2). Strong experience with data reconciliation frameworks (using complex SQL queries) that perform various recon checks per industry standards to ensure migrated data is functionally equivalent. Good understanding of Schema Evolution Management strategies and enforcement approaches. Extraction Types Used Kafka, ANSI SQL, FTP, Apache Spark Data Formats JSON, Avro, Parquet Platforms Hadoop (HDFS/Hive), Snowflake, Apache Iceberg, Sybase IQ

Collected experience requirement

6-9 years