Senior Staff - IT Data Engineering and Analytics

Job Purpose and Responsibilities Position Purpose: The Senior Data Engineer (Forward Deployed) is a hands-on engineer who works alongside the business, not behind a ticket queue. You will embed with trading, operations, finance and risk teams to understand how they work, turn their questions into precise source-to-target mappings, and ship production Databricks pipelines on the StoneX Unified Data Platform in days and weeks, not quarters. AI is a core part of how you work: you use it to analyse sources, draft mappings, generate and test code, and build reusable frameworks that make the whole Global Data Services team faster. The role suits an engineer who enjoys a live problem, a demanding stakeholder and a fast pace, and who can explain a technical trade-off as clearly to a trader as to an engineer. What 'Forward Deployed' means here: Embedded: you spend a large share of your time with business users - in their meetings, their tools and their data - and you own the outcome, not just the code. End to end: discovery → mapping → pipeline → validation → adoption, delivered by one accountable engineer. Fast and iterative: working increments in front of users early, with short feedback loops. Leverage: every engagement leaves something reusable behind (a framework, a template, an agent) so the next one is faster. Technology Ecosystem: Data platform: Databricks (Delta Lake, Unity Catalog, Lakeflow / Workflows, Databricks SQL), Python, PySpark, SQL, SQL Server Integration: RedPanda (Kafka), OpenAPI / REST APIs, Airflow, Profisee MDM, Microsoft Dynamics AI-assisted delivery: Claude Code, GitHub Copilot, Databricks Assistant and AI/BI Genie Engineering: medallion architecture (Bronze / Silver / Gold), data warehousing, CI/CD (GitHub Actions, Jenkins), Docker, Kubernetes Consumption and cloud: Power BI, React.js; Microsoft Azure Primary duties will include: Discover and translate: run discovery sessions with business stakeholders; turn loosely defined asks into clear requirements, data contracts and acceptance criteria; play your understanding back in plain language. Map: produce rigorous source-to-target mappings (STTM) covering field-level lineage, transformation logic, keys, defaults, SCD handling and data quality rules, signed off by engineers and business users alike. Build: design and build batch and streaming pipelines on Databricks (PySpark, SQL, Delta Lake, Unity Catalog, Lakeflow) that follow the medallion patterns of the Unified Data Platform. Accelerate with AI: use AI coding agents every day to profile sources, draft STTMs, and generate pipelines and tests; build reusable AI-enabled frameworks such as metadata-driven ingestion, STTM-to-code generation and automated data quality and reconciliation checks. Validate and land: reconcile outputs with the business, own data quality and monitoring, and drive adoption in Power BI and downstream systems. Operate: own production reliability for what you build (CI/CD, alerting, incident response, root-cause analysis), including cost and performance tuning on shared Databricks compute. Communicate: give crisp written and verbal updates to business and technology leaders, record decisions in Confluence and Jira, and reset expectations early when scope, dates or data change. Raise the bar: coach engineers on business engagement and AI-assisted delivery; review others' AI-generated code for correctness, security and performance. Qualifications: To land this role you will need: 6-9 years of hands-on data engineering, including 3+ years building production pipelines on Spark / Databricks. 2+ years in a business-facing delivery role (forward-deployed, consulting, solutions or embedded analytics engineering) where you owned outcomes with non-technical stakeholders. Proven ability to write source-to-target mappings and dimensional / medallion data models from ambiguous requirements. Expert Python, PySpark and SQL; strong Delta Lake, Unity Catalog and Databricks Workflows / Lakeflow experience on Microsoft Azure. Recent, demonstrable use of AI coding agents to deliver production work, with the judgment to verify, test and reject what they produce. Excellent verbal and written communication; confident running workshops and explaining trade-offs to senior business stakeholders. Thrives at pace: comfortable with shifting priorities, tight deadlines and overlap with US / UK business hours when needed. What makes you stand out: Capital markets, commodities or trade-lifecycle data experience (trades, positions, postings, balances, reference data). You have built a reusable framework or internal tool, ideally AI-enabled, that other engineers adopted. Streaming with Kafka / RedPanda, and APIs (OpenAPI / REST) as data sources. MDM (Profisee) and CRM (Microsoft Dynamics) data integration. Power BI semantic models or Databricks AI/BI (Genie) spaces that business users rely on. Time in a start-up, consulting firm or forward-deployed engineering team. Education / Certification Requirements: Bachelor's degree in Computer Science, Engineering, Mathematics or a related field, or equivalent demonstrable experience. Databricks Certified Data Engineer Professional (strongly preferred) or Associate. Azure Data Engineer Associate or equivalent. Generative AI certifications (e.g., Databricks Generative AI Engineer Associate) are a plus. Working environment: Hybrid (as per RTO policy), Bengaluru. Embedded with business and technology teams across the US, UK, India and Poland; regular overlap with US / UK hours. Travel: occasional, for discovery workshops and go-lives.