Lead Backend Engineer - Data Engineering & AI

Design, build, and maintain the backend services, APIs, data-platform automation, and AI agents behind our data products. This is a backend-heavy role: distributed Python services, streaming LLM/agent runtimes on AWS, RAG pipelines, and the dbt/Airflow automation that powers our data products. Responsibilities Design, build, and maintain backend services and APIs in Python — RESTful and streaming (SSE) endpoints, agent runtimes on AWS Bedrock AgentCore / ECS Fargate , and event-driven Lambda handlers. Architect service boundaries and data flows : define contracts between services, model persistence, and manage state, caching, and asynchronous/background processing. Build RAG pipelines and tool-calling AI agents over our data products: retrieval, orchestration, grounding, and evaluation. Design data access and storage layers : schema/data modeling, query performance, connection/session management, and integration with warehouses (Snowflake) and key-value stores (DynamoDB). Implement auth and identity : OAuth/OIDC flows (per-user 3LO, token vaulting, session binding), least-privilege IAM, secrets management. Build and maintain data-platform automation : dbt models, MWAA/Airflow orchestration, and tooling for discoverable, governed, consumable data products. Own service reliability and delivery : Terraform, GitHub Actions CI/CD, container builds, structured logging, metrics/tracing, alerting, and cost controls. Set technical direction: system and API design, code review, and mentoring. Required skills Backend engineering 5+ years designing, building, and maintaining production backend services at scale. 10+ years if Lead level engineering candidate. Expert-level Python for server-side development; solid grasp of at least one web/async framework (e.g. aiohttp, FastAPI, Flask) and the WSGI/ASGI model. Service and API design : REST (and/or gRPC), request/response and streaming patterns, pagination, versioning, idempotency, and backward-compatible contracts. Data layer : SQL and data modeling, query optimization and indexing, transactions, connection pooling; experience with relational, warehouse (Snowflake), and NoSQL/key-value (DynamoDB) stores. Server-side patterns : caching strategies, background jobs/workers, queues and event-driven processing, rate limiting, retries/backoff, and timeouts. Performance & reliability : profiling, load handling, latency/throughput trade-offs, graceful degradation, and designing for failure. Observability : structured logging, metrics, distributed tracing, and debugging live production issues. Software engineering fundamentals Object-oriented programming (required) : encapsulation, abstraction, inheritance, composition, polymorphism; SOLID principles ; design patterns applied pragmatically; strong domain modeling. Solid data structures & algorithms ; ability to reason about time/space complexity. Concurrency and async programming (async/await, threading, event loops) and their failure modes. Testing (unit, integration, end-to-end) and testable design; Git and PR-based workflows; disciplined code review. Cloud & infrastructure Production AWS : ECS/containers, Lambda, IAM, API Gateway, DynamoDB. Infrastructure as code with Terraform ; CI/CD (GitHub Actions or equivalent) and container builds. Distributed systems Building services that are horizontally scalable, resilient, and loosely coupled; handling consistency, retries, idempotency, and partial failure. Security OAuth/OIDC , authn/authz, token handling, least-privilege access, multi-tenant isolation, secrets management. AI / LLM engineering Building LLM applications in production (not research). RAG : chunking, embeddings, vector search, hybrid search, reranking, grounding/citations, context-window management, retrieval evaluation. Agents : prompt engineering, tool use/function calling, structured outputs, agent orchestration (single- and multi-step), prompt caching. Integration with model providers — Anthropic/Claude, AWS Bedrock/AgentCore, Snowflake Cortex — and response streaming. AI quality & ops : eval harnesses, guardrails, tracing/observability, token/latency/cost optimization. AI security : prompt injection, data exfiltration, PII handling, per-user identity/RBAC enforcement. Nice to have dbt , Airflow/MWAA , Snowflake . MCP (Model Context Protocol) and/or RAG Slack platform (Bolt, Socket Mode, Block Kit) or other real-time/conversational backends. Fine-tuning/adaptation, semantic caching, or model routing/fallback. Experience adding AI capabilities to existing production systems.