QA Engineer – Automation & Agentic Applications

Role overview

QA Engineer – Automation & Agentic Applications at Beroe

Design, develop, and maintain automated test frameworks and test scripts using Python.

We are looking for a QA Engineer with strong automation expertise and experience testing intelligent/agentic applications to join our growing engineering team.

Responsibilities

• You will play a critical role in ensuring the quality, safety, and reliability of our platform by developing automated test. frameworks, validating agent autonomy and decision-making, and collaborating closely with developers, product owners, and DevOps.
• Write and execute manual and automated test cases to validate product features, agent behavior, and functionality.
• Integrate automated tests into CI/CD pipelines to ensure continuous quality and fast feedback.
• Collaborate with developers and product managers to identify, reproduce, and resolve defects early in the lifecycle.
• Define and document test plans, strategies, and acceptance criteria for both deterministic and agentic features.
• Perform scenario-based, exploratory, and chaos testing to validate agent decision making under uncertainty, failure, and edge conditions.
• Verify observability and explainability of agent behavior (logs, traces, intermediate actions) to ensure compliance, transparency, and auditability.
• Test human-in-the-loop workflows, ensuring proper escalation, overrides, and user safety.
• Conduct root cause analysis of production issues, including agent misbehavior, and contribute to long-term fixes.
• Contribute to the development and enforcement of QA standards and best practices.
• Must-Have Qualifications 5+ years of experience in QA Engineering with a focus on test automation.
• Proficient in Python for writing test scripts and tools.
• Hands-on experience with testing frameworks such as Pytest, Selenium, or Playwright.
• Strong understanding of software testing methodologies, including unit, integration, system, and end-to-end testing.
• Familiarity with REST APIs and tools like Postman, Swagger, or Bruno for API testing Solid understanding of Agile/Scrum methodologies.
• Strong problem-solving skills with the ability to think beyond “happy path” testing.
• Nice-to-Have Qualifications Experience testing agentic or AI-driven applications (e.g., autonomous decision making, multi-step workflows) Experience with scenario-based or chaos testing frameworks Familiarity with observability and monitoring tools (e.g., Datadog, Grafana, AWS CloudWatch) to validate agent behavior Experience testing microservices Experience with CI/CD tools like GitHub Actions, Jenkins, or GitLab CI Knowledge of security and guardrail testing (role-based access, safe action validation)

Employment details

• Location: India - Remote

Additional role details

• This role goes beyond traditional QA: you will be working with AI-driven, agentic applications that can plan, act, and adapt.
• Your job will be to make sure they behave correctly, safely, and predictably under real-world conditions.