QA & Validation Agent — Engineering Guide

InfinitySDLC Engineering Guides · 05/12

A QA agent should produce evidence that a change works, not merely more test code. It combines change-impact analysis, risk-based test selection, controlled execution and machine-verifiable grading.

The difficult question is not whether an agent can make a test pass. It is whether the test would reject the wrong behavior, whether it ran against the exact candidate and whether missing evidence can stop a release.

Reference engineering design. Sections 5.1–5.5 and the implementation blueprint follow the Enterprise AI Agent Mesh handbook. Sections 5.6–5.12 extend the design with proposed production controls and linked technical sources. Scenarios and numerical examples are illustrative, not completed client deployments or measured Infinity results.

The agent can propose tests and explain failures. It must not be able to redefine the acceptance criteria simply to obtain a green result.

The Header diagram is a conceptual pipeline. Its pass rate, coverage, defect counts and percentage changes are illustrative placeholders, not client results or release thresholds. Open the diagram at full size.

5.1 Data and tools

• Git MCP: diffs, ownership, changed symbols and test mappings.
• Test catalog RAG: historical tests, flaky-test metadata, defects, coverage maps and contract schemas.
• CI MCP: run jobs, rerun with bounded parameters and retrieve JUnit reports, coverage and artifacts.
• Environment MCP: request an ephemeral environment and an approved synthetic dataset.
• Observability MCP: query traces, logs and metrics produced by the candidate build.

These are internal capability contracts, not a claim that each vendor supplies an identical MCP server. Read access, test execution and draft patch creation are separate permissions. A QA task does not require production deployment authority.

5.2 Test generation is downstream of impact analysis

First compute what can break. Map changed symbols to callers, contracts, data schemas, feature flags and incident history. Select or generate tests against those risks, rather than asking a model to increase the test count.

The handbook favors the smallest existing test layer that proves the required behavior: unit before integration, contract before full end-to-end, and deterministic replay before manual UI automation. This is a selection principle, not permission to omit integration evidence when the failure crosses a boundary.

change_impact:
  changed_symbols: [Invoice.calculate_tax]
  downstream_contracts: [BillingAPI.v3.Invoice]
  historical_defects: [BUG-912, INC-2026-144]
  risk_tags: [money, rounding, locale]
validation_plan:
  - existing: test_tax_rounding_matrix
  - generated: contract_invoice_v3_tax_precision
  - replay: incident_2026_144_payload_set

The identifiers above are illustrative handbook examples. In a deployment, each reference must resolve to a versioned requirement, contract or reproducible defect fixture.

5.3 Use coding agents inside a disposable branch

Claude Code or Codex can inspect implementation, propose tests and execute a suite in an ephemeral worktree or branch. Supply no production credentials. Require a patch, commands and exit codes, structured results and unresolved risks—not a message saying that testing is complete.

Apply the same principle to a local-model harness. The enterprise control plane owns tools, identity and evidence rules regardless of provider. OpenAI’s Codex deployment guidance distinguishes the technical sandbox boundary from approval policy and describes controlled outbound networking. These are independent controls, not alternatives to a careful prompt.

Test code is executable untrusted code. Package installation, test fixtures and build hooks can also execute commands. Run them under the intended resource and network restrictions, with synthetic or approved masked data.

5.4 Flakiness and false confidence

• Track pass and fail history per test. A flaky test passing once is weaker evidence than a stable, reproducible result.
• Where feasible, run high-risk generated tests against a known-bad revision or targeted mutation to establish that they can detect the defect.
• For natural-language or UX evaluation, combine an LLM judge with deterministic assertions and calibration against human-labeled examples.
• Do not let a model invent the requirement interpretation and serve as the only judge of its own correctness.

Keep original failures and retries visible. “Eventually green” and “passed on the first attempt” are different outcomes, and the release policy should retain that distinction.

5.5 Release evidence packet

The handbook’s packet connects a specific candidate with its required gates, observations and artifact references. The following values preserve its illustrative example; they are not a release currently under review.

Quarantine cannot silently satisfy a required gate. A production packet must identify whether the quarantined test is outside the required set, replaced by equivalent approved evidence or covered by a valid exception. Likewise, a small performance delta needs workload and measurement context before it can support a release decision.

5.6 Keep the test oracle independent of the implementation

An oracle defines what correct behavior means. A test that calculates its expected value by calling the same implementation under test can reproduce the bug rather than detect it. As a production extension, record the source of each consequential expectation: an approved rule, contract, reviewed fixture or independent reference calculation.

Freeze the relevant acceptance criteria and reference baseline before test generation. Permit the agent to propose a change to them, but require separate review. Protect CI gate definitions, required-test selectors and golden fixtures from silent edits made solely to pass validation.

Illustrative rounding example

Assume a deliberately simplified test rule: multiply a decimal amount of 19.99 by 0.075, then round once to two decimal places using half-up rounding. The reviewed expected result is 1.50. A truncation mutation returns 1.49 and should fail the targeted assertion. This is a synthetic software example, not a tax rule or financial recommendation.

Store that expected result in a reviewed fixture or calculate it with an independent reference implementation. Record the rule and rounding stage. Do not infer a universal property such as “tax on the total equals the sum of line taxes”: that can fail when the approved rule rounds per line.

The same discipline applies outside arithmetic. Access-control tests need an authoritative permission matrix; migration tests need declared invariants; UI tests need an approved behavior or visual baseline. A screenshot is not its own approval.

5.7 Make incomplete impact analysis visible

A test-impact graph is an optimization aid, not an exhaustive proof of safety. Preserve how each edge was obtained—static analysis, runtime coverage, contract metadata or incident mapping—and the revision and coverage limits of its source.

A test covering a symbol on yesterday’s main branch does not necessarily exercise today’s changed feature-flag path. Bind selection to the candidate revision, environment, relevant configuration and flag state. Track unmapped files, unsupported languages, external consumers and dynamic relationships explicitly.

Distinguish “not selected,” “not collected,” “skipped,” “failed” and “infrastructure error.” They answer different questions. Time or cost limits may stop execution, but the incomplete work must remain visible to the release gate.

5.8 Prove that a test can fail for the right reason

A negative check should produce the failure predicted by the requirement: the wrong rounding result, an unauthorized response or a duplicate business effect. A mutation run that fails because its container cannot start has not demonstrated the assertion’s sensitivity.

Use targeted mutations for high-consequence paths before paying for broad mutation runs. Retain the mutation identity, expected failing assertion, actual failure and the corresponding passing candidate run. Keep baseline health visible so a pre-existing failure is not credited to the new test.

Stryker’s equivalent-mutant guidance explains why some mutations leave behavior unchanged and cannot be killed by a discriminating test. A surviving mutant therefore needs classification; a universal 100% mutation-score target can be misleading. Record excluded or equivalent cases and their review basis.

Properties and metamorphic checks

Propose properties from explicit business invariants, not from generic expectations about software. A serialization round trip should preserve the fields promised by its contract. Repeating an operation should avoid a second business effect only where idempotency is part of that operation’s definition. Input-order invariance is appropriate only when order is semantically irrelevant.

Keep preconditions, fixture versions, random seeds and minimized failing examples with the result. Passing generated properties provides evidence within the tested domain; it does not establish correctness for every possible input.

5.9 Treat reproducibility, retries and quarantine as policy

Define the test environment sufficiently to interpret a failure: candidate digest, test revision, dependency lock, runtime or browser version, operating system, locale, timezone, fixture revision and relevant feature flags. Capture differences between the coding workspace and CI instead of calling both simply “the same test.”

Playwright distinguishes tests that pass initially, fail initially but pass on retry, and remain failed. Preserve attempt-level results in the evidence packet. Repeated retries consume a predefined budget; the agent cannot increase that budget indefinitely or select only the successful attempt for reporting.

Quarantine needs an owner, reason, expiry and compensating evidence where the test protects a required risk. A new failure in a historically flaky test is still an observation worth investigating. Separate nondeterministic product behavior from infrastructure instability before assigning responsibility.

For visual comparisons, pin the environment used to create and validate the baseline. Playwright notes that rendering varies with operating system, browser, fonts and other environment factors. Record viewport and browser details; wait for relevant assets and fonts; control known nondeterministic regions with a reviewed policy. Do not mask the very behavior under test.

An agent may propose a new snapshot, but changing the image does not approve it. A reviewer or a separately defined acceptance process must decide whether the new appearance is intentional.

5.10 Produce evidence that the agent cannot self-approve

Separate the coding workspace from the service that evaluates release gates. The agent proposes test changes and dispatches authorized jobs. A controlled collector obtains run identity and results from CI, associates artifacts with the exact candidate and applies a versioned policy outside the model.

1. 1. Freeze candidate and risks
2. 2. Execute controlled tests
3. 3. Collect attributed evidence
4. 4. Apply release policy

A file hash proves that bytes match a recorded digest; it does not prove that those bytes represent a trustworthy test. Validate the job, workflow revision, producer and relationship to the candidate. Keep protected independent checks for critical behavior: a test author may otherwise change both the implementation and the evidence generator.

Do not give untrusted test jobs the release signer or privileged deployment token. GitHub’s secure-use guidance warns about the impact of compromised runners and recommends least-privilege token permissions. A QA branch must not be able to rewrite the policy it is being graded against.

# Illustrative production evidence contract
validation_record:
  candidate_digest: verified-build-digest
  test_revision: reviewed-test-commit
  risk_manifest: approved-risk-manifest-digest
  environment_manifest: verified-environment-id
  policy_revision: release-gates-r7
  required_gate_ids: [unit, contract, migration]
  results:
    contract:
      outcome: blocked
      reason: verification_missing_for_target_version
      ci_run_id: authoritative-run-reference
  artifact_refs: [report-id, trace-bundle-id]
  unresolved_risks: [consumer-version-compatibility]
  release_decision: not_authorized

Require reconciliation between selected tests, collected tests and reported outcomes. Pytest uses exit code 5 when no tests were collected. A wrapper that ignores that result must not turn an empty suite into release evidence. Missing reports, incomplete shards and cancelled runs also remain incomplete, not implicitly successful.

5.11 Verify compatibility and performance in their actual context

A contract test only supports a claim about the contract and versions it verified. Record consumer and provider versions, the tested interaction and the deployment combination being evaluated. A passing check for an obsolete consumer cannot establish compatibility with an unverified one.

Pact’s pending-pact behavior deliberately allows some new unsupported contracts to avoid breaking the provider build while still recording failed verification. A green provider job is therefore not proof that a new consumer is safe to deploy. Consult the version-specific verification outcome and the applicable deployment compatibility check.

Keep runtime observations reproducible

For performance evidence, preserve workload mix, request count, concurrency, warm-up, data size, environment capacity, baseline revision and error treatment. A p95 delta without those conditions can describe different experiments rather than a regression.

Use an approved comparison policy and disclose uncertainty, insufficient samples and infrastructure changes. Do not silently discard slower runs. A load smoke test can catch obvious breakage without establishing the full production SLO.

The QA agent supplies this evidence to the Risk & Reliability Agent and the Change & Release Orchestration Agent. It does not waive an operational gate because its own test suite looks convincing.

5.12 Evaluate the QA agent against misleading success

The proposed acceptance suite below extends the handbook’s quality gates. It tests whether the agent and its surrounding services reject plausible but invalid evidence.

Grade patch correctness, defect detection, risk coverage, artifact attribution and permission compliance separately. For subjective output, use a written rubric, human-labeled calibration examples and deterministic checks around the judge. Anthropic’s evaluation guidance distinguishes the transcript from the resulting environment state and discusses combining grading methods. The final authority here is a verified outcome, not a persuasive test summary.

Measure detection on labeled historical or seeded defects, false rejection of known-good candidates, reproducibility, reviewer corrections and time to a complete evidence packet. Keep development fixtures separate from held-out evaluations and disclose their sizes and change classes. More generated tests, higher coverage or a green dashboard are not substitutes for those measurements.

The operating package should include a versioned risk manifest, independent oracles, execution receipts, negative-check results, unresolved risks and the policy decision. That is what makes the QA agent useful to an engineering team rather than merely productive at generating code.

Implementation Blueprint: Prove the Change

Build a test-impact graph

(:Symbol)-[:CALLED_BY]->(:Symbol)
(:Test)-[:COVERS]->(:Symbol)
(:ContractTest)-[:VALIDATES]->(:ApiSchema)
(:Defect)-[:CAUSED_BY]->(:Symbol)
(:FeatureFlag)-[:CONTROLS]->(:CodePath)

Populate the graph from static analysis, coverage, CI history and incident mappings. Select existing tests first and generate new ones for uncovered risks. Preserve the source blueprint’s separation between impact analysis and test generation; add provenance and coverage limitations rather than treating every edge as equally authoritative.

Claude Code / Codex task contract

Goal: add the minimum tests required for RISK-17.
Workspace: ephemeral worktree at <sha>.
Allowed:
  - scoped repository read/write
  - tests in the isolated runner
  - package manager through an allowlisted proxy
Forbidden:
  - production systems and credentials
  - arbitrary outbound network access
Required output:
  - changed_files[]
  - commands[] with exit_codes
  - tests_added[]
  - unresolved_risks[]
  - targeted negative/mutation evidence

For high-risk generated tests, preserve evidence that the supplied mutation or known-bad revision fails for the intended reason. The production extensions above add protected oracle and gate configuration, independent collection and candidate-level attribution to this source contract.

Quality gates from the handbook

• Generated tests run in CI, not only in the local coding harness.
• High-risk assertions receive a negative or mutation check where practical.
• Snapshot changes cannot be accepted automatically merely because output changed.
• Flaky tests cannot be the only release evidence.
• Fixtures use synthetic or approved masked data.

Sources and shared prerequisites

Adapted from Article 5 and Blueprint 5 of the September 2026 Enterprise AI Agent Mesh handbook. The production extensions are proposed engineering choices, not claims that a client deployment has already passed them. Links to OpenAI, Playwright, Stryker, pytest, Pact, GitHub and Anthropic support the specific platform behaviors discussed above.

The Enterprise Agent Platform Foundation defines shared identity, policy, retrieval, sandboxing, approvals and audit. The Environment Agent supplies the verified test environment. Pin and validate actual runtime, tool and model versions before implementation.

Explore the series

Previous: Environment Agent

Series foundation: Enterprise Agent Platform Foundation

Next: Change & Release Orchestration Agent