Senior QA Engineer (AI Systems & Automation)
About the position Essential Software Inc. is a trusted partner to federal agencies, including the National Cancer Institute (NCI), delivering secure, cloud-based platforms that support large-scale cancer data and biomedical research. As a Senior QA Engineer (AI Systems & Automation), you will lead quality strategy and test automation for critical data platforms and AI-powered experiences. You will ensure both traditional software and AI/agentic systems are reliable, explainable, and safe in a federal, mission-driven environment. You will: Own end-to-end quality for complex web, API, data, and AI/ML-powered features Design AI-aware test strategies and automation that leverage GenAI and agentic frameworks Mentor QA engineers and collaborate closely with cross-functional teams and government partners Responsibilities • Develop and maintain test plans, test cases, traceability, and test data for product and AI features • Execute manual and automated tests for web applications, RESTful APIs, data workflows, and AI/ML features • Own automated regression suites, release readiness criteria, and provide clear go / no-go quality signals • Participate in agile ceremonies, validate end-to-end functionality, and ensure user stories (including AI features) meet acceptance criteria • Manage the full defect lifecycle, including triage, prioritization, root cause analysis, and verification of fixes • Maintain QA documentation, runbooks, and quality dashboards • Design and execute test strategies for AI/LLM-powered capabilities, including virtual agents, chatbots, copilots, and RAG-based systems • Use LLM-powered tools (e.g., ChatGPT, Claude, Copilot) to accelerate test design, data generation, exploratory testing, and script authoring • Build and refine QA-focused AI agents that can: Scrape UI and verify DOM structures Validate data against backend or ground-truth sources Auto-generate and maintain test scripts Run self-correcting / autonomous test flows • Evaluate and integrate agentic frameworks (e.g., OpenAI Assistants API, AWS Bedrock Agents, LangGraph, MCP) into QA workflows • Define and monitor AI-specific quality metrics (accuracy vs. ground truth, hallucination and error rates, safety / policy adherence) • Ensure AI and virtual agent experiences are accurate, consistent, and high quality in a federal context • Plan and execute performance, load, and scalability testing (e.g., JMeter or equivalent) • Validate data integrity and transformation quality across complex biomedical data pipelines and AI-enhanced workflows • Partner with engineers and data scientists to ensure AI/ML models and integrations are testable, observable, and measurable post-deployment • Mentor QA team members in both traditional and AI-augmented QA practices • Collaborate with development, DevOps, product, UX, and data teams to improve testability, shift-left quality, and increase automated coverage • Integrate automation into CI/CD (e.g., GitHub Actions, Jenkins, Azure DevOps, GitLab CI), monitor test health and flakiness, and address coverage gaps • Communicate quality risks, trends, and mitigation plans to technical and non-technical stakeholders, including government partners Requirements • Bachelor’s degree in computer science, Information Technology, Engineering, or related field • 5+ years of software QA experience (manual and automation) in production environments • 2+ years providing technical or process leadership (e.g., lead QA, primary product QA owner, mentor, or manager) • Strong experience with UI automation tools (Selenium WebDriver, Playwright, or Cypress) • Experience testing RESTful APIs and microservices architectures • Hands-on experience integrating automated tests into CI/CD pipelines (GitHub Actions, Jenkins, Azure DevOps, or GitLab CI) • Professional proficiency in Python or JavaScript for test automation • Hands-on use of GenAI tools (e.g., ChatGPT, Claude, Copilot) for QA tasks such as test-case generation, data creation, and exploratory testing • Understanding of AI/agentic concepts: Tool-calling / function invocation Multi-step / chain-of-thought workflows Autonomous / self-healing test flows AI-driven data comparison and validation • Experience with performance / load testing (e.g., JMeter or equivalent) • Proficiency with Jira or similar issue tracking tools • Strong written and verbal communication skills, including the ability to explain AI-related quality risks to stakeholders • Ability to prioritize, multitask, and operate effectively in complex, mission-driven environments Nice-to-haves • AWS Cloud Practitioner certification • Experience with modern automation stacks (Playwright or Cypress) and API testing tools (Postman, REST-assured, pytest, or similar) • Experience testing AI/ML-powered features (LLM applications, RAG systems, agents, recommendation engines, or chatbots) • Experience with one or more: LangChain or LangGraph AWS Bedrock Agents or OpenAI Assistants API MCP (Multi-Context Protocol) or similar orchestration frameworks • Experience designing or testing internal QA copilots or automation bots for test authoring or execution • Familiarity with test management tools (e.g., TestRail, Zephyr) • Knowledge of accessibility standards (WCAG) and basic security testing practices • Prior QA experience in healthcare, life sciences, biomedical informatics, or other regulated data environments • ISTQB or similar certification Benefits • Competitive benefits • Professional development opportunities • Collaborative, supportive culture Apply tot his job