QA & Validation Agent — Engineering Guide

InfinitySDLC Engineering Guides · 05/12

Reference implementation guide, not a report of a completed client deployment. Code, configurations, metrics and policies are illustrative. Adapt and validate them before production use.

A QA agent should produce evidence that a change works, not merely more test code. The agent combines change-impact analysis, risk-based test selection, controlled execution and machine-verifiable grading.

5.1 Data and tools

• Git MCP: diff, file ownership, changed symbols, test mapping.
• Test catalog RAG: historical tests, flaky-test metadata, defects, coverage maps, contract schemas.
• CI MCP: run jobs, rerun with parameters, retrieve JUnit/coverage/artifacts.
• Environment MCP: request ephemeral test environment and synthetic dataset.
• Observability MCP: query traces/logs/metrics generated by the candidate build.

5.2 Test generation is downstream of impact analysis

First compute what can break. Map changed symbols to callers, contracts, data schemas, feature flags and incident history. Then generate or select tests that cover those risks. The agent should prefer modifying the smallest existing test layer that proves behavior: unit before integration, contract before full end-to-end, deterministic replay before manual UI automation.

change_impact:
  changed_symbols: [Invoice.calculate_tax]
  downstream_contracts: [BillingAPI.v3.Invoice]
  historical_defects: [BUG-912, INC-2026-144]
  risk_tags: [money, rounding, locale]
validation_plan:
  - existing: test_tax_rounding_matrix
  - generated: contract_invoice_v3_tax_precision
  - replay: incident_2026_144_payload_set

5.3 Use coding agents inside a disposable branch

Claude Code or Codex can generate tests and fixtures effectively when they can inspect the implementation and execute the test suite. Put them on an ephemeral worktree/branch with no production credentials. Their success criterion is not “test file created”; it is green execution plus mutation/negative checks where appropriate. The test agent should return a patch, commands executed and structured result summary.

5.4 Flakiness and false confidence

• Track historical pass/fail distributions per test. A newly passing flaky test does not provide the same evidence as a stable deterministic test.
• Run generated tests against both the candidate and a known-bad mutation when feasible; this detects tests that cannot fail.
• For LLM-graded UX or natural-language behavior, use multiple deterministic assertions around the LLM judge and calibrate the judge on human-labeled examples.
• Never let the same model both generate a requirement interpretation and be the sole judge of correctness without external evidence.

5.5 Release evidence packet

Implementation Blueprint: Prove the Change

Build a test-impact graph

(:Symbol)-[:CALLED_BY]->(:Symbol)
(:Test)-[:COVERS]->(:Symbol)
(:ContractTest)-[:VALIDATES]->(:ApiSchema)
(:Defect)-[:CAUSED_BY]->(:Symbol)
(:FeatureFlag)-[:CONTROLS]->(:CodePath)

Populate edges from static analysis, coverage, CI history and incident mappings. The agent selects existing tests first and generates new tests only for uncovered risks. This is materially better than prompting a coding model to “write tests for this PR.”

Claude Code / Codex task contract

Goal: add the minimum tests required for RISK-17.
Workspace: ephemeral worktree at <sha>.
Allowed: repo read/write, tests, package manager through allowlisted proxy.
Forbidden: production systems, arbitrary network, secrets.
Required output: changed_files[], commands+exit_codes, tests_added[], unresolved_risks[], and evidence that high-risk generated tests fail against a supplied mutation/known-bad revision.

Quality gates

• Generated tests run in CI, not just the local harness.
• High-risk assertions get a negative/mutation check where practical.
• Snapshot changes cannot be auto-accepted merely because output changed.
• Flaky tests cannot be the only release evidence.
• Fixtures use synthetic or approved masked data.

Source and Shared Prerequisites

Adapted from the September 2026 Enterprise AI Agent Mesh handbook, Article 5 and Blueprint 5. The Enterprise Agent Platform Foundation guide provides the shared identity, MCP, retrieval, sandbox, audit and evaluation design, plus the source handbook’s further-reading list. Validate model, protocol and tool versions before production use.

Explore the Series

Previous guide · Series foundation · Next guide