Senior Data Engineer

Role Overview We are building a bespoke data platform focused on Financial Crime transaction analysis. The platform processes tens of millions of transaction records daily and enables large-scale analytics to identify suspicious patterns and behavioral anomalies. We are looking for a hands-on Data Engineer with strong Python and Spark expertise who can design scalable data pipelines, perform large-scale data analysis, and contribute to the architecture of a high-volume data platform. This role is ideal for engineers who enjoy deep coding, distributed data processing, and building custom data platforms rather than relying purely on managed tools. Key Responsibilities Build and maintain scalable data pipelines across stages such as data download, ingestion, and analysis within a bespoke data platform. Develop high-performance data processing logic using Python and PySpark. Perform large-scale transaction data analysis using custom algorithms and in-house logic to detect financial crime patterns. Work with high-volume datasets (80-90 million records processed regularly) and optimize pipeline performance. Design efficient data models and transformations for large-scale processing. Write optimized SQL queries on PostgreSQL (RDS), leveraging: Window functions Partitioning Query performance optimization Work with Delta tables and Parquet-based storage formats for efficient data processing. Build and maintain Spark batch and streaming jobs. Implement engineering best practices including unit testing, static code analysis, and CI/CD practices. Contribute to data platform architecture and system design decisions. Required Skills Core Engineering Skills Strong Python programming expertise Deep understanding of: Functional programming Object-Oriented Programming (OOP) Design patterns Python execution and invocation mechanisms Hands-on experience with Apache Spark / PySpark Strong SQL expertise including: Window functions Partitioning Query optimization Experience building end-to-end data pipelines Data Platform Engineering Experience designing scalable data processing architectures Strong understanding of distributed data processing Familiarity with standard data pipeline engineering practices, including: Unit testing Static code analysis (Sonar or similar) Code linting Pipeline reliability and monitoring Good to Have AWS exposure (especially EMR and S3) Experience with Apache Airflow for job orchestration and scheduling Exposure to Kafka or streaming architectures Familiarity with Databricks environments (as upstream data source) Experience working in banking, financial services, or financial crime analytics Tech Stack Languages: Python, SQL Processing: Apache Spark / PySpark Storage: Delta Tables, Parquet Database: PostgreSQL (RDS) Orchestration: Airflow (good to have) Cloud: AWS (EMR, S3 - good to have) Code Quality: Sonar, Unit Testing, Linting What We're Looking For Strong hands-on engineer, who can do more that using standard ETL tools. Comfortable working with large-scale datasets Ability to design and build scalable data platform components Someone who enjoys solving complex data engineering problems through code