Blue hexagonal agent nodes around a governance core, shared platform artwork for the Operations engineering guide.

Operations Agent: Durable Execution, Process Conformance, and the Connective Tissue of the Mesh

Infinity Technologies
Enterprise Agent Mesh Engineering Guides
September 2026
No items found.

The table of content

Enterprise Agent Mesh · Engineering Guide 07 / 12

An Operations Agent should not be a long-lived chat loop pretending to be a workflow engine. Durable state, retries, approvals, idempotency and compensation belong outside the model; the model is called for bounded judgement steps.

Source: Article 7 of The Enterprise Agent Mesh — Building Twelve Production AI Agents on Claude Code, September 2026, physical pages 146–170. This web edition preserves the chapter’s twelve-section structure. Explicitly labelled web-edition qualifications distinguish current documentation checks and implementation hardening from the handbook. Reference-deployment figures, thresholds and timings are illustrative, not verified Infinity client results or universal operating limits. The cover is shared agent-platform artwork, not a screenshot of an Operations deployment.

On this page

1. What this agent is actually for

Operations is the follow-through layer of the mesh. A supply-chain disruption becomes an incident with a communications obligation. A procurement exception becomes a change request. A support escalation starts an SLA clock. The agent’s value is not a new domain model; it is making cross-system work finish without losing the state that everyone assumes somebody else is tracking.

The handbook frames seven primary workflows: incident context assembly, runbook execution with timeline capture, postmortem drafting, SLA/OLA breach prediction, process-conformance checking, change-request impact assessment, and the weekly operating-review pack. A long tail of recurring follow-through sits beside them: reconcile a CMDB owner after a leaver, close the linked issue when a change reaches its terminal state, or chase an approval that has stalled.

WorkflowAgent contributionAuthority boundary
Incident context packBounded CI graph, recent changes, alert signatures, related incidents and evidence gaps.Read-only; no diagnosis is treated as fact.
Runbook executionDry-run, step proposal, step-scoped approval, execution, verification and timeline note.Never batch or invent steps; live execution depends on blast radius and authenticated approval.
Incident communicationsDraft impact, scope, action and next-update time from the live record and contractual obligation.Sev1 customer-facing publication requires a named human approver.
Process conformanceExplain deterministic deviations from a Petri-net/BPMN-style process model.The model explains; a deterministic alignment decides where the deviations are.
CAB impact assessmentIndependent blast-radius and collision analysis from CMDB/change evidence.A human owns approval of the change.
Weekly operating reviewTrace material variances to named cases and mechanisms.“Timing” or “volume” without evidence is not an explanation.

The autonomy ladder is deliberately asymmetric. Drafting postmortems and customer communications can remain L1; approved reversible runbook actions can reach L2; ticket enrichment and explicitly reversible follow-through may reach L3 after tool-specific evidence; nightly read-only conformance scans and context-pack preparation can reach L4. The handbook’s promotion windows and numeric gates are reference policy, not a default policy for every operator.

The agent never declares an incident Resolved, Closed or Cancelled. It never publishes Sev1 customer communication without a named human approver, never crosses a change freeze without an authorised emergency change, and never owns blast-radius 4 execution.

Those are capability boundaries, not etiquette. Resolution is intentionally absent from the ordinary tool surface. The model may propose severity changes or next actions, but the authoritative state transition remains human-owned.

2. Architecture: build for orchestration, not for reasoning

An LLM turn is a poor home for forty minutes of control flow. A process can die after a side effect landed but before the transcript recorded it. A retry can then repeat the external action. Context compaction can retain the interesting hypothesis and discard the unglamorous fact that a customer update was already sent. The system therefore needs durable state outside the agent session.

The chapter’s reference design uses Temporal. A deterministic workflow owns state, timers, retry policies, signals and the compensation stack. Side-effecting work runs in activities with stable idempotency keys derived from (workflow_id, step_id). The workflow calls Claude for a bounded judgement task, receives structured output, and then decides how the process advances. The model never receives a tool that can start, signal or terminate the durable workflow itself.

PagerDuty / Datadog / ITSM events
               |
               v
      Durable IncidentWorkflow
   state · timers · retries · sagas
       /        |          \
      v         v           v
context pack   bounded      execute step
(read-only)    agent plan   activity
      |         |           |
      +---------+-----------+
                |
        evidence/effect log
                |
          human approval
                |
       customer / ITSM effects

This inversion is the main architecture decision: the workflow remembers; the model judges. A proposal activity may get eight to fourteen turns. It does not own an unbounded 200-turn incident lifecycle.

Compensation follows naturally. Each runbook step declares whether it is compensable and, if so, names the compensating step. If a later step fails, the workflow unwinds the recorded stack in reverse. Non-compensable effects—an email, a propagated DNS change, a schema migration—are explicit points of no return and should require a stronger human confirmation before crossing them.

If Temporal is not part of the target stack, the handbook describes a smaller alternative: a Postgres workflow state table plus an effect table with a unique idempotency key. That can still move the record of what happened outside the agent process. It is less feature-rich, but it preserves the essential property: restarting the model cannot erase or duplicate completed effects.

Web-edition qualification. Temporal’s current documentation describes durable, crash-resilient execution with replayable workflow state, signals, timers and retryable activities. Its Python SDK exposes execute_activity, wait_condition, workflow signals and retry policies. Treat the code below as a reference excerpt with application dependencies omitted; validate determinism, activity semantics and versioning against the installed SDK before deployment.

3. Repository layout and CLAUDE.md

operations-agent/
  CLAUDE.md
  .mcp.json
  .claude/
    agents/
      incident-context-assembler.md
      runbook-driver.md
      conformance-analyst.md
      ops-review-narrator.md
    skills/
      incident-comms-draft/SKILL.md
      weekly-operating-review/{SKILL.md,reference.md}
      runbook-execution-protocol/SKILL.md
    hooks/
      hooks.json
      ops-policy-gate.py
      freeze-calendar.py
  workflows/
    incident.py
    sla_watch.py
    recurring.py
    activities/{context.py,runbook.py,comms.py}
  runbooks/
    rb-0142-payment-gateway-degraded.yaml
  process_models/
    incident-management.pnml
  mcp/
    ops/server.py
    runbook/server.py
    pm/server.py
  evals/
    golden/{context_packs.jsonl,runbook_steps.jsonl,conformance.jsonl}
    chaos/injections.yaml
    rubrics/handoff_quality.md

Runbooks live as structured, reviewable YAML in version control, not only as prose in a wiki. A step that is an addressable record can carry its blast radius, dry-run behaviour, post-condition, compensation and verification logic. A wiki can remain the rendered view, but it should not be the executable source of truth.

CLAUDE.md carries invariants that are true on every run and expensive to violate. Long runbook catalogues, process models and account-level SLA matrices stay outside it.

# Operations Agent — operating invariants

## Absolute
- Never set an incident to Resolved, Closed or Cancelled.
- Never publish Sev1 customer communication without a named human approver.
- If tool state contradicts your recorded state, escalate_and_stop.

## Time
- Emit UTC ISO-8601 timestamps with explicit offsets.
- Refuse naive timestamps; do not guess a timezone.
- Order incident timelines by event time, not ingestion time.

## Severity
- Human authority sets severity; the agent may propose a change with evidence.

## Clocks
- The customer SLA outranks an internal OLA.
- Pause a clock only when the authoritative record says it is paused.

## Blast radius
- 0 read-only
- 1 reversible, single non-production CI
- 2 reversible, production, single CI
- 3 multiple CIs or customer-visible
- 4 irreversible or data-affecting
- Radius 3 requires two distinct human approvers.
- Radius 4 is never agent authority.

The exact severity taxonomy, account calendars and blast-radius definitions must be owned by operations/change management and versioned as policy. The source values are a reference model. A deployment should not hide organisation-specific change rules inside the prompt merely because the prose is convenient.

4. Subagents: narrow context, narrow tools

The chapter uses four roles. None both plans and executes. That separation is enforced through tool availability and backend identity, not only prompt text.

incident-context-assembler

---
name: incident-context-assembler
description: Assemble a bounded, deduplicated context pack for an active incident.
tools: mcp__ops__assemble_incident_context, mcp__itsm__search_changes, mcp__corpus__search
model: haiku
effort: low
maxTurns: 8
disallowedTools: Bash, Write, Edit, WebFetch
permissionMode: default
memory: project
background: false
color: cyan
---
Call assemble_incident_context exactly once.
Report truncated_sections verbatim.
Return, in order:
1. blast radius now
2. recent changes
3. alert signatures ordered by first_seen
4. at most five similar incidents, each with its resolution
5. open and related incidents/problems
Do not diagnose. If the context tool fails twice, escalate_and_stop.

Boundedness is part of correctness. A user must know when a section was truncated. Filling the missing area with a model summary defeats the whole character-budget design.

runbook-driver

---
name: runbook-driver
description: Drive one matched runbook step at a time against a live incident.
tools: mcp__runbook__get_runbook, mcp__runbook__execute_runbook_step, mcp__itsm__add_work_note, mcp__ops__escalate_and_stop
model: sonnet
effort: medium
maxTurns: 14
disallowedTools: Bash, Write, Edit, WebFetch, mcp__statuspage__publish_update
permissionMode: default
skills: runbook-execution-protocol
memory: project
color: orange
---
For each step:
1. fetch it and restate blast_radius and compensable
2. dry-run first; report proposed commands verbatim
3. obtain approval for this step, not a blanket prior approval
4. execute using the workflow-supplied idempotency key
5. check the post-condition
If verification fails, do not retry or continue: escalate_and_stop.
Write a work note after every executed step; the work note is the timeline.

The model never generates its own idempotency key. Stability across retries and replays is a workflow property; letting the model invent the key destroys it.

conformance-analyst

The conformance analyst does not decide whether a case conformed. A deterministic alignment tool compares the event log with the process model and returns fitness plus deviations—moves present in the log but not allowed at that point, and model-required moves that did not occur. The model then reads only the local evidence around those deviations and classifies likely causes such as control_gap, tooling, judgement, data_quality or training.

This is a useful pattern beyond ITSM: let deterministic process mining find where to look; use the language model to explain why the deviation happened and whether it mattered. The process model is authoritative for “should”; the event log is authoritative for “did.”

ops-review-narrator

The weekly narrator receives a governed operating pack and explains material variances. It can use a higher-reasoning model because the task is sparse and review-heavy, but it has no execution path. It must attach named cases and a mechanism to a variance or label it “not explained.” Avoid granting a background narrator operational tools simply because it can see the same metrics as the incident agent.

Web-edition subagent qualification. Current Claude Code documentation gives subagents their own context and tool restrictions. Plugin-defined subagents have different support for hooks, MCP configuration and permission modes than project subagents. Validate where each definition lives before assuming every frontmatter field in this reference repository is enforced the same way.

5. Skills: encode the procedure, not the current state

incident-comms-draft

---
name: incident-comms-draft
description: Draft incident communications from the live incident record.
allowed-tools: mcp__itsm__get_incident, mcp__ops__assemble_incident_context, mcp__corpus__search
disallowed-tools: mcp__statuspage__publish_update
user-invocable: true
arguments: [incident_id, audience]
---
# Incident communications draft
Audience is internal, account or public.

1. What is happening — observable impact in the customer's terms.
2. Who is affected — service scope, or say scope is not established.
3. What is being done — present tense; no internal CI names for public.
4. Next update — absolute UTC time from the contracted obligation.

Do not attribute cause before the postmortem.
Do not infer an ETA from a similar incident.
For Sev1, leave approved_by empty for a named human.

The current freeze posture and notification obligations should be injected from deterministic wrappers or tools at invocation time. They are live state, not prompt facts. A “next update” timestamp is a contractual commitment; it comes from the account obligation, not the model’s guess about what sounds reasonable.

The weekly operating-review skill uses the same progressive-disclosure pattern. Keep the main skill small: section order, variance materiality rules, the requirement to trace an explanation to named cases, and the explicit option to write “not explained.” Put metric definitions, account thresholds and worked prose examples in a companion reference loaded only when that section is needed.

Runbook execution is also a skill because it is a repeatable protocol. It should teach step-at-a-time behaviour, dry-run/approval/verification ordering and escalation rules. The runbook’s actual executable facts belong in the structured runbook record, so a skill release cannot silently change a production command.

6. Integrations and MCP contracts

SystemReference integrationEngineering constraint
ServiceNow / CMDBGateway-wrapped REST and shaped server-side endpoints.Use the actual service identity’s ACLs; keep write scope narrow to work notes/proposals.
Jira Service ManagementIssue search, comments and governed transitions.Do not treat a generic transition endpoint as permission to close authoritative incident state.
PagerDuty / OpsgenieIncident/log reads, notes and governed escalation.Normalise timestamps at the adapter boundary.
Datadog / GrafanaMetrics/events/monitor state.Bound queries and preserve source time and aggregation.
Slack / TeamsOperator UI, incident channel and approval surface.Mirror decisions/effects to durable evidence; chat retention is not the incident ledger.
Status page / communicationsPublication endpoint behind human approval.Sev1 publication is never an unconstrained agent tool.
Runbook storeGit-backed structured YAML.Read steps by immutable version; execute through a guarded adapter.
TemporalDurable workflow host.Not exposed to the model as MCP; application code calls the model.

Vendor rate limits in the handbook are reference observations, not current contractual guarantees. Slack, for example, documents tier-specific limits and returns Retry-After on HTTP 429; chat.postMessage is generally limited around one message per second per channel. The Events API expects fast acknowledgement and retries failed deliveries. A production adapter should queue work, respect current headers and reconcile delivery, rather than bake a table value into the prompt.

ServiceNow’s REST APIs apply ACLs and role checks for the authenticated identity. This is exactly why tests under an engineer’s account are insufficient evidence for an agent service principal. For context packs that need several joins, use a bounded server-side shape where appropriate rather than dumping multiple unbounded table reads into the transcript.

Context assembly owns its own budget

BUDGET = {
  "affected_ci_graph": 1500,
  "recent_changes": 3000,
  "alert_history": 2500,
  "similar_incidents": 4000,
  "open_related": 1000,
}

def collapse_alerts(alerts):
    signatures = {}
    for alert in alerts:
        key = (alert.monitor_id, alert.ci, alert.transition)
        row = signatures.setdefault(key, {"signature": alert.title, "count": 0,
            "first_seen": alert.ts, "last_seen": alert.ts})
        row["count"] += 1
        row["first_seen"] = min(row["first_seen"], alert.ts)
        row["last_seen"] = max(row["last_seen"], alert.ts)
    return sorted(signatures.values(), key=lambda r: r["first_seen"])

The tool collapses an alert storm before it reaches the context window and reports dropped rows per section. A generic maximum-result-size hint is a backstop. Section budgets and deterministic deduplication are the actual design.

Execution defaults to dry-run in the handler

async def execute_runbook_step(args):
    # Reference adapter excerpt; dependencies are application services.
    dry_run = bool(args.get("dry_run", True))
    step = await runbooks.step(args["runbook_id"], args["step_id"])
    missing = [ci for ci in step.cis if not await cmdb.exists(ci)]
    if missing:
        return err(f"runbook stale; missing CIs: {missing}")
    if int(args.get("blast_radius", -1)) != step.blast_radius:
        return err("declared blast radius differs from the runbook")
    if not dry_run:
        approval = await approvals.verify(args.get("approval_token"), step)
        if not approval.valid:
            return err(approval.reason)
        if step.blast_radius >= 3 and len({s.principal for s in approval.signers}) < 2:
            return err("two distinct approvers required")
    prior = await effects.get(args["idempotency_key"])
    if prior:
        return ok({**prior.result, "replayed": True})
    result = await executor.run(step, dry_run=dry_run)
    if not dry_run:
        await effects.put(args["idempotency_key"], result)
    return ok(result)

The important property is not the exact Python. Safety defaults live in application code, approvals are verified against authenticated records, and the effect log is checked before execution. A JSON schema default or prompt instruction is not enough.

7. Retrieval: incidents are noisy, runbooks decay, process is an event log

The corpus contains runbooks, incidents and postmortems, change records, architecture decision records, operating-model documentation and SLA/OLA schedules. It should not become a raw Slack archive or a credential graveyard. Customer-sensitive names and postmortem content remain behind entitlement filters enforced in the retrieval service.

Chunking follows the object being reasoned about. Runbooks are chunked by executable step, because the step is the unit of dry-run, approval, execution and verification. Postmortems are chunked by section. Change records keep structured fields as fields and embed only meaningful implementation/backout text.

Similar incidents: remove the template before embedding

Ticket text is dominated by boilerplate—form headers, integration-user lines, alert payloads and repeated status text. Embedding it makes “similar” mean “uses the same ticket template.” The reference design first strips stable template blocks, then classifies remaining content into symptom narrative, diagnosis, resolution and boilerplate. It embeds symptom and resolution separately.

Before vector search, filter candidates using structured signals such as service, affected-CI class, controlled symptom type and explicit error code. Then query the symptom index to answer “have we seen this?” and the resolution index to answer “what fixed cases that look like this?” Fuse, rerank and return a small evidence set. A similar incident without its resolution is trivia, not guidance.

Runbook retrieval is health-aware

A runbook can be semantically relevant and operationally dangerous because the environment changed around it. Maintain a health record using signals such as last_verified_at, CMDB drift and recent execution failure rate. Penalise stale runbooks; exclude them entirely once they cross a reviewed staleness limit, and route them to a decay report for their owner.

The source’s 0.80 exclusion line, 31% unverified-in-twelve-months observation and 9% decommissioned-CI observation are reference-deployment figures. They illustrate why staleness belongs in retrieval ranking; they are not thresholds or performance claims for a new installation.

Process mining is not prose retrieval

The event log is ground truth for what happened. A model asked to infer process from documentation gives you the documented process—the gap between documented and actual behaviour is exactly what conformance analysis needs. Use deterministic optimal alignment between the event sequence and the process model, then pass the resulting deviation windows to the model for explanation.

PM4Py’s current documentation provides Petri-net alignment algorithms including A*-style/state-equation and Dijkstra variants. The choice of model, cost function and event normalization is part of the governed process-mining implementation. The LLM should not silently invent alignment costs or a process model because the library can calculate them.

8. Guardrails: stop must be a first-class outcome

The operations policy gate protects four classes of boundary: authoritative incident state, customer communication, production execution and change freezes. Backend identity and tool absence remain the primary controls; hooks add a fast, auditable policy layer.

# Reference policy logic, simplified
if tool == "update_incident" and state in {"resolved", "closed", "cancelled"}:
    hard_block("a named human owns terminal incident state")
if customer_facing and severity == 1 and not authenticated_approver:
    hard_block("Sev1 customer communication requires a named human approver")
if executing_runbook_step:
    if blast_radius > context_ceiling:
        deny("blast radius exceeds this context's ceiling")
    if change_freeze_active and not authorized_emergency_change:
        hard_block("active freeze requires an authorised emergency change")
    if blast_radius >= 3 and distinct_approvers < 2:
        hard_block("two-person rule")

Web-edition hook correction. The handbook comments that a JSON deny may be overridden by a later allow. Current Claude Code hook documentation says matching PreToolUse decisions are combined using the most restrictive result: deny > ask > allow. Exit code 2 still blocks the tool call. Treat the current documented semantics as authoritative and test the installed release; do not carry the older “later allow overrides deny” comment into production.

A hook is not the whole authorization system. The service principal must lack terminal-state or broad production credentials. An approved step carries the human identity through the gateway, and the target system enforces that human’s entitlements. Someone who cannot restart a service directly cannot grant an agent more power through a chat card.

escalate_and_stop

The less obvious guardrail is a tool that lets the agent admit it has lost the state. Triggers include repeated idempotency conflicts, a tool result contradicting recorded workflow state, failed compensation, or a context pack whose truncation hides the CI under investigation. The tool writes a handoff pack, notifies the incident commander and moves the durable workflow into an explicit human-handoff state.

The handoff pack says what was executed, what definitely landed, what may be half-done, what evidence conflicts, and the single next thing the human should check. “Stop cleanly” is a production capability. An agent without it tends to improvise precisely when the environment is least predictable.

9. Production deployment: workflow state is durable, agent state is disposable

The following reference excerpt preserves the handbook’s topology while making the separation explicit. Activity implementations, approval validation, compensation logic and the Claude adapter are application code and are omitted here.

from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy

@workflow.defn
class IncidentWorkflow:
    def __init__(self):
        self.approved = {}
        self.compensations = []

    @workflow.signal
    def approve(self, step_id: str, token: str):
        self.approved[step_id] = token

    @workflow.run
    async def run(self, event: dict):
        pack = await workflow.execute_activity(
            assemble_context, event["incident_id"],
            start_to_close_timeout=timedelta(minutes=4),
            retry_policy=RetryPolicy(maximum_attempts=3))
        plan = await workflow.execute_activity(
            propose_plan, {"pack": pack, "incident_id": event["incident_id"]},
            start_to_close_timeout=timedelta(minutes=6),
            retry_policy=RetryPolicy(maximum_attempts=2))
        for step in plan["steps"]:
            sid = step["step_id"]
            if step["blast_radius"] >= 1:
                await workflow.execute_activity(
                    request_approval, step,
                    start_to_close_timeout=timedelta(minutes=1))
                await workflow.wait_condition(
                    lambda: sid in self.approved,
                    timeout=timedelta(minutes=20))
            stable_key = f"{workflow.info().workflow_id}:{sid}"
            result = await workflow.execute_activity(
                run_step,
                {"step": step, "idempotency_key": stable_key,
                 "approval_token": self.approved.get(sid)},
                start_to_close_timeout=timedelta(minutes=10),
                retry_policy=RetryPolicy(maximum_attempts=4))
            if result.get("compensable"):
                self.compensations.append((sid, result["compensating_step"]))

Temporal workflow code must remain deterministic under replay. Do not generate a random idempotency key inside the activity boundary and then expect replay to recover it. Persist or derive keys from durable workflow identity and step identity. Treat changes to timer/condition behavior as versioned workflow changes where required by the SDK’s determinism rules.

The Claude activity is deliberately bounded

async def propose_plan(input_data):
    # Reference excerpt. Host configures actual SDK types and structured parsing.
    options = {
        "model": "approved-worker-model",
        "allowed_tools": ["mcp__runbook__get_runbook", "mcp__corpus__search",
                          "mcp__ops__escalate_and_stop"],
        "disallowed_tools": ["mcp__runbook__execute_runbook_step",
                             "Bash", "Write", "Edit", "WebFetch"],
        "max_turns": 12,
    }
    return await call_agent_for_structured_plan(input_data, options)

The planner cannot execute. Execution lives in another activity with another capability set. This is stronger than asking the same session to “be careful” after it has persuaded itself that urgency justifies a shortcut.

Retries live on activities with explicit idempotency contracts. A timeout after a possibly completed write is not automatically safe to repeat; authoritative readback or the effect log determines whether the effect exists. Budget/turn caps are circuit breakers: if the bounded planner cannot converge, the correct state is human handoff, not a larger autonomous loop.

Verification boundary. The reference Python snippets above are structurally consistent with the current Temporal Python workflow APIs and are presented as excerpts, not as a runnable repository. Production requires workflow/activity registration, data converters, exception typing, compensation implementation, SDK version pinning and tests against a sandbox target system.

10. Evaluation: chaos-test the control plane

Operations evaluation cannot stop at answer quality. The core claim is that workflows survive crashes without duplicate external effects and that humans receive a usable handoff when the agent stops. The reference evaluation set therefore includes context packs, runbook-step labels, process-conformance cases and mid-incident handoff snapshots.

MetricWhat it testsHandbook reference gate
Causal-change recall@3The known causal change appears among the top three changes in the context pack.At least 0.85; hard fail.
Pack-truncation honestyEvery non-empty truncated_sections is surfaced.1.000; hard fail.
Runbook-step accuracyProposed next step matches senior-engineer labels.At least 0.90; hard fail.
Stale-runbook selectionSelected runbooks above the reference staleness band.No more than 0.02; hard fail.
Conformance kappaDeviation-cause classification vs process analyst.At least 0.75; warning.
Handoff qualityExecuted state, confirmed-vs-assumed, next check and timestamp clarity.At least 2.5/3; hard fail.
Duplicate-effect rateMore than one external effect for one idempotency key under injected failures.0.000; hard fail.
Resume-success rateWorkflow completes after worker kill/restart.At least 0.99; hard fail.

These values describe the handbook’s reference programme. A new deployment should derive acceptance thresholds from its own incident severity, contractual obligations and failure costs. What should remain invariant is the shape of the test: a side-effecting orchestration agent needs chaos and recovery evaluation, not only a prompt-quality judge.

Kill it at the awkward point

The chaos suite runs against sandbox ITSM/PagerDuty-style endpoints, kills the worker at an activity boundary, during an activity before the effect, and after the effect but before the activity result is recorded. Restart it and assert: the workflow completes, each idempotency key produces at most one effect, and no compensation remains dangling.

The source reports that this style of test found a random UUID generated inside the wrong boundary, a non-deduplicated Slack post and a non-idempotent compensation. Those are reference failure observations, not Infinity client incidents. Recreate the failure classes in your own sandbox and make the gate deterministic.

MTTA and MTTR belong in operational trend analysis, not a CI prompt gate. They are confounded by incident mix, staffing and changing classification. Compare service lines and longer windows; do not claim the model improved MTTR because one quarter moved after deployment.

11. Failure modes and their controls

Duplicate side effects after retry

A status page or work-note update lands, the activity crashes before recording success, and the retry emits it again. Use a stable idempotency key, check the effect log before execution and record the result under a unique constraint. Exactly-once business effects are a property of the adapter plus store, not a property of a model transcript.

A stale runbook executes confidently

The step text is perfect but names a CI retired fourteen months ago. Penalise stale runbooks during retrieval and assert that every named CI resolves in the CMDB immediately before execution. Do not “helpfully” map an old hostname to its replacement inside the model.

Alert-storm context flooding

Thousands of downstream alerts cause the model to anchor on the loudest symptom. Collapse alerts into signatures inside the context tool and order by first-seen evidence rather than count. The handbook’s 3,400-alert example collapsing to six signatures demonstrates the pattern; it is an illustrative reference observation.

The timeline is reconstructed in the wrong timezone

One system renders local instance time, another emits UTC, and the postmortem reverses causal order by minutes. Normalize at the MCP boundary, require timezone-aware timestamps and carry a tz_normalised marker. Refuse a naive timestamp rather than guessing the zone.

Local OLA optimization breaks the customer SLA

A routing action can look beneficial to one queue while consuming the global contractual clock. Put the objective hierarchy in the operating invariants and make routing tools return the effect on customer clocks, not only local backlog.

The organisational failure: the agent subsidises preventable work

If the agent becomes cheap and competent at recurring defects, teams can stop fixing the defect. Classify closed work into genuine incident, known-defect-not-fixed, process gap, should-be-self-service and data-quality categories, attach a cost, and report the avoidable categories to the service owner. Automation should reduce the source of operational demand, not merely make the queue more tolerable.

All failure examples in this guide come from the supplied reference handbook or from explicitly labelled web-edition hardening. They are engineering test cases, not claims about incidents or measured outcomes at Infinity customers.

12. Build order: earn write authority only after recovery works

Week 1: stand up durable workers, the effect log with a unique idempotency constraint, read-only context assembly across ITSM/PagerDuty/observability, and the evidence timeline. Execute nothing. Measure whether the bounded pack is useful enough that on-call engineers stop opening the same tabs.

Month 1: convert runbooks to step-level structured YAML in git, add staleness scoring and the first decay report, keep runbook-driver dry-run-only, introduce L1 communications drafting, and run conformance checking on one documented process. Build the chaos suite before enabling a live runbook step.

Quarter 1: enable carefully selected blast-radius-1 execution at L2 with step-scoped approval and the two-person rule wired for larger actions, add CAB impact assessment, the weekly operating-review pack and “tickets that should not exist” reporting. The handbook explicitly keeps L3 off until the chaos suite is clean across two releases.

The permanent architecture rule remains: the workflow engine owns state, the model owns bounded judgement, deterministic tools own measurements and alignments, and humans own consequential state transitions. Increasing model capability does not change those responsibilities.

Primary sources and web-edition notes

The structural source is Article 7 of the supplied Enterprise Agent Mesh handbook, with the shared platform from Part 0. The links below support the current implementation qualifications. They do not turn the reference excerpts into a complete vendor integration.

Continue the Enterprise Agent Mesh series

← Previous: 06 Finance Risk Agent — Evidence Assembly, Adversarial Review, and Model Risk Management

01 Recruitment Agent · 02 HR Agent · 03 Supply Chain Agent · 04 Procurement Agent · 05 Finance Agent

Next: 08 Customer Support Agent — Containment Quality, Not Deflection Rate. Forthcoming.

Infinity Technologies
Enterprise Agent Mesh Engineering Guides
September 2026
No items found.

Recent Insights

Product Discovery Agent: customer signals and analytics feed hypothesis development, product drafts and human review.
September 2026

Product Discovery Agent — Engineering Guide

Engineering guide 02/12: turn customer feedback, product analytics and repository context into evidence-backed hypotheses and traceable requirements.
Infinity Technologies
InfinitySDLC Engineering Guides
Technologies
Planning and Architecture Agent: approved requirements, constraints and policies lead to design alternatives, architecture records and delivery plans reviewed by a human architect.
September 2026

Planning & Architecture Agent — Engineering Guide

Engineering guide 03/12: convert approved requirements into architecture decisions, dependency-aware delivery plans and machine-checkable work packages.
Infinity Technologies
InfinitySDLC Engineering Guides
Technologies
Environment Agent reference workflow linking Git, Infrastructure as Code, CI/CD and isolated environments. Secret references and approved execution remain controlled by platform services.
September 2026

Environment Agent — Engineering Guide

Engineering guide 04/12: build an Environment Agent for reproducible infrastructure, bounded Kubernetes diagnostics and disposable test environments.
Infinity Technologies
InfinitySDLC Engineering Guides
Technologies
Conceptual QA and Validation pipeline: risk-based tests produce independently verified evidence for an approval gate. Dashboard numbers are illustrative, not client results.
September 2026

QA & Validation Agent — Engineering Guide

Engineering guide 05/12: build a QA agent that selects risk-based tests, uses isolated coding agents and produces verifiable release evidence.
Infinity Technologies
InfinitySDLC Engineering Guides
Technologies
Change and Release Orchestration pipeline covering impact analysis, risk, approval, staged delivery, monitoring and rollback decisions.
September 2026

Change & Release Orchestration Agent — Engineering Guide

Engineering guide 06/12: coordinate change approval, CI/CD, progressive delivery and rollback with deterministic state transitions and two-phase writes.
Infinity Technologies
InfinitySDLC Engineering Guides
Technologies
Observability Agent reference workflow: bounded logs, metrics, traces and change evidence feed competing hypotheses and a human-owned remediation handoff.
September 2026

Observability Agent — Engineering Guide

Engineering guide 07/12: correlate traces, metrics, logs and deployments using bounded telemetry queries and evidence-backed competing hypotheses.
Infinity Technologies
InfinitySDLC Engineering Guides
Technologies
Security operations loop showing preventive controls feeding detection, response, recovery and approved post-incident learning.
September 2026

Security Prevention Agent — Engineering Guide

Engineering guide 08/12: combine deterministic security scanners, threat-model RAG and contextual code review before merging software changes.
Infinity Technologies
InfinitySDLC Engineering Guides
Technologies
Threat Detection Agent in a security operations loop: preventive controls feed detections, bounded analysis and approved incident-response actions.
September 2026

Threat Detection Agent — Engineering Guide

Engineering guide 09/12: enrich SIEM and EDR alerts, resolve entities and build evidence-backed incident timelines without unbounded containment powers.
Infinity Technologies
InfinitySDLC Engineering Guides
Technologies
Risk and Reliability Agent combines service topology, SLO posture, change context and verified resilience evidence to produce reviewable risk decisions and bounded experiments.
September 2026

Risk & Reliability Agent — Engineering Guide

Engineering guide 10/12: quantify change risk using SLOs, error budgets, dependency graphs and resilience evidence rather than an ungrounded model score.
Infinity Technologies
InfinitySDLC Engineering Guides
Technologies
Incident Response coordinates triage, approved containment, recovery and post-incident learning within a wider security operations loop.
September 2026

Incident Response Agent — Engineering Guide

Engineering guide 11/12: build an incident copilot with structured state, specialist-agent handoffs, typed runbooks and human-approved mitigation.
Infinity Technologies
InfinitySDLC Engineering Guides
Technologies
AI model routing control plane showing policy-first eligibility across hosted coding harnesses and local open-weight inference, followed by evaluation, capacity and audit controls.
September 2026

AI Model Router Agent — Engineering Guide

Engineering guide 12/12: route tasks across Claude, Codex and self-hosted models using data policy, capabilities, evaluation scores, cost and availability.
Infinity Technologies
InfinitySDLC Engineering Guides
Technologies
Governed enterprise agent mesh: connected hexagonal agents around a protected platform core.
September 2026

Recruitment Agent: Evidence Assembly Under a High-Risk Regulatory Regime

Engineering guide 01/12: build a Recruitment Agent that assembles requirement-linked evidence, preserves provenance and keeps candidate decisions with humans.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
Technologies
Governed enterprise agent platform connecting specialist agents around a protected control core.
September 2026

HR Agent: Effective-Dated, Jurisdiction-Scoped Policy Retrieval

Engineering guide 02/12: build an HR Agent that resolves employee context before retrieval, answers against effective-dated policy and routes sensitive cases.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
Technologies
Shared enterprise agent platform illustration used for the Supply Chain Agent engineering guide.
September 2026

Supply Chain Agent: Exception Narratives Over an Optimiser You Already Own

Engineering guide 03/12: build a Supply Chain Agent that triages planning exceptions, explains shortage causality with provenance and delegates quantities to deterministic solvers.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
Technologies
Governed enterprise agent platform with specialized nodes around a protected control core.
September 2026

Procurement Agent: Segregation of Duties Encoded in the Tool Layer

Engineering guide 04/12: build a Procurement Agent where approvals, supplier banking and payment authority are structurally outside the model’s tool and credential boundary.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
Technologies
Enterprise Agent Mesh platform illustration for the Finance Agent engineering guide.
September 2026

Finance Agent: Numbers From Tools, Never From the Model

A production engineering guide to a Finance AI Agent where every figure comes from deterministic tools and immutable fact packs, while the model is limited to grounded narrative and workflow orchestration.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
Technologies
Shared agent-platform illustration for the Finance Risk engineering guide: connected hexagons surrounding a governance core.
September 2026

Finance Risk Agent: Evidence Assembly, Adversarial Review, and Model Risk Management

Engineering guide 06/12: deterministic treasury calculations, facility-specific covenant definitions, adversarial challenge and human decision authority.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
Technologies
Shared Enterprise Agent Mesh illustration for the Customer Support Agent engineering guide.
September 2026

Customer Support Agent: Containment Quality, Not Deflection Rate

Engineering guide 08/12: containment quality, account-scoped retrieval, guarded customer sends, escalation packets and evidence-driven support autonomy.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
Technologies
Shared Enterprise Agent Mesh illustration for the Sales Agent engineering guide.
September 2026

Sales Agent: Make the CRM True, Then Worry About Selling

Engineering guide 09/12: make CRM state evidence-backed before generating selling assistance, with governed field updates, customer commitments and cross-agent handoffs.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
Technologies
Shared Enterprise Agent Mesh illustration for the Marketing Agent engineering guide.
September 2026

Marketing Agent: Generation Is the Commodity, the Constraint System Is the Product

Engineering guide 10/12: make generation subordinate to market-scoped claims, rights, channel rules, attribution discipline and measurable brand controls.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
Technologies
Shared Enterprise Agent Mesh illustration for the Legal Agent engineering guide.
September 2026

Legal Agent: The Playbook Is the Program

Engineering guide 11/12: build a Legal Agent where executable playbooks, matter access, contract lineage and privilege boundaries govern model-assisted review.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
Technologies
Enterprise Agent Mesh platform illustration for the Compliance Agent guide, with blue hexagonal agents and a central governance shield.
September 2026

Compliance Agent: Evidence Logistics, Control Testing, and the Mesh’s Control Plane

Engineering guide 12/12: build a Compliance Agent for reproducible control testing, sealed evidence, curated crosswalks and continuous mesh conformance.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
Technologies
Enterprise agent platform illustration: a layered control core connects specialized agents, knowledge, models, telemetry and isolated execution.
September 2026

Enterprise Agent Platform Foundation — Engineering Guide

Engineering guide 01/12: build the shared control plane, MCP gateway, ACL-aware retrieval, isolated runtimes and evaluation system for an enterprise AI agent mesh.
Infinity Technologies
InfinitySDLC Engineering Guides
Technologies
Fuzzy logic model of usability of websites of higher education institutions in the context of digitalization of educational services

Fuzzy logic model of usability of websites of higher education institutions in the context of digitalization of educational services

Fuzzy logic model for evaluating university website usability and improving digital experience in higher education
Business
Management
Technologies
Tools
EU countries clustering for the state of food security using machine learning techniques

EU countries clustering for the state of food security using machine learning techniques

The study applies clustering methods to group EU countries by food security levels and develop tailored policy recommendations.
Technologies
Time Series Forecasting of Agricultural Product Prices Using Elman and Jordan Recurrent Neural Networks

Time Series Forecasting of Agricultural Product Prices Using Elman and Jordan Recurrent Neural Networks

A study of Elman and Jordan neural networks for predicting historical agricultural prices using time series data.
Business
Technologies
Identifying Stock Market Crashes by Fuzzy Measures of Complexity

Identifying Stock Market Crashes by Fuzzy Measures of Complexity

This article explores how fuzzy logic and complexity measures can help detect early signals of stock market crashes, offering a more stable and insightful alternative to traditional analysis methods.
Business
Tools
Management
Technologies
Fuzzy Clustering: A Data-Driven Revolution in ESG Portfolio Strategy

Fuzzy clustering approach to portfolio management considering ESG criteria: empirical evidence from the investment strategies of the EURO STOXX Index

Find out more about the new ESG portfolio strategy using fuzzy clustering to align sustainability with strong, adaptive performance.
Technologies
Fuzzy Cluster Analysis: Smarter Trade Timing with Real Profitability

Identifying Moments of Decision Making on Trade in Financial Time Series Using Fuzzy Cluster Analysis

how Fuzzy Cluster Analysis enhances trading strategies by combining technical indicators with probabilistic clustering and financial performance metrics
Technologies
Tools
Secure and Accelerate Software Development with InfinitySecOps™

Infinity Technologies Introduces InfinitySecOps™

A Revolutionary 11-Stage DevSecOps Framework
Technologies
Tools
Introducing InfinityFrame™: A Paradigm Shift in Tech Design and Software Architecture

Introducing InfinityFrame™: A Paradigm Shift in Tech Design and Software Architecture

The Evolving Challenge of Software Architecture
Technologies