Blue hexagonal agent nodes around a governance core, shared platform artwork for the Operations engineering guide.

Operations Agent: Durable Execution, Process Conformance, and the Connective Tissue of the Mesh

AI/ML
About the Task
Reference architecture for an Operations Agent that coordinates long-running cross-system work without making an LLM the workflow engine.
results
Design outcome: durable execution, idempotent side effects, step-scoped approvals, conformance evidence and explicit human handoff.
results
Reference design and illustrative controls, not a report of measured client outcomes.
Services used
No items found.

The table of content

Enterprise Agent Mesh · Engineering Guide 07 / 12

An Operations Agent should not be a long-lived chat loop pretending to be a workflow engine. Durable state, retries, approvals, idempotency and compensation belong outside the model; the model is called for bounded judgement steps.

Source: Article 7 of The Enterprise Agent Mesh — Building Twelve Production AI Agents on Claude Code, September 2026, physical pages 146–170. This web edition preserves the chapter’s twelve-section structure. Explicitly labelled web-edition qualifications distinguish current documentation checks and implementation hardening from the handbook. Reference-deployment figures, thresholds and timings are illustrative, not verified Infinity client results or universal operating limits. The cover is shared agent-platform artwork, not a screenshot of an Operations deployment.

On this page

1. What this agent is actually for

Operations is the follow-through layer of the mesh. A supply-chain disruption becomes an incident with a communications obligation. A procurement exception becomes a change request. A support escalation starts an SLA clock. The agent’s value is not a new domain model; it is making cross-system work finish without losing the state that everyone assumes somebody else is tracking.

The handbook frames seven primary workflows: incident context assembly, runbook execution with timeline capture, postmortem drafting, SLA/OLA breach prediction, process-conformance checking, change-request impact assessment, and the weekly operating-review pack. A long tail of recurring follow-through sits beside them: reconcile a CMDB owner after a leaver, close the linked issue when a change reaches its terminal state, or chase an approval that has stalled.

WorkflowAgent contributionAuthority boundary
Incident context packBounded CI graph, recent changes, alert signatures, related incidents and evidence gaps.Read-only; no diagnosis is treated as fact.
Runbook executionDry-run, step proposal, step-scoped approval, execution, verification and timeline note.Never batch or invent steps; live execution depends on blast radius and authenticated approval.
Incident communicationsDraft impact, scope, action and next-update time from the live record and contractual obligation.Sev1 customer-facing publication requires a named human approver.
Process conformanceExplain deterministic deviations from a Petri-net/BPMN-style process model.The model explains; a deterministic alignment decides where the deviations are.
CAB impact assessmentIndependent blast-radius and collision analysis from CMDB/change evidence.A human owns approval of the change.
Weekly operating reviewTrace material variances to named cases and mechanisms.“Timing” or “volume” without evidence is not an explanation.

The autonomy ladder is deliberately asymmetric. Drafting postmortems and customer communications can remain L1; approved reversible runbook actions can reach L2; ticket enrichment and explicitly reversible follow-through may reach L3 after tool-specific evidence; nightly read-only conformance scans and context-pack preparation can reach L4. The handbook’s promotion windows and numeric gates are reference policy, not a default policy for every operator.

The agent never declares an incident Resolved, Closed or Cancelled. It never publishes Sev1 customer communication without a named human approver, never crosses a change freeze without an authorised emergency change, and never owns blast-radius 4 execution.

Those are capability boundaries, not etiquette. Resolution is intentionally absent from the ordinary tool surface. The model may propose severity changes or next actions, but the authoritative state transition remains human-owned.

2. Architecture: build for orchestration, not for reasoning

An LLM turn is a poor home for forty minutes of control flow. A process can die after a side effect landed but before the transcript recorded it. A retry can then repeat the external action. Context compaction can retain the interesting hypothesis and discard the unglamorous fact that a customer update was already sent. The system therefore needs durable state outside the agent session.

The chapter’s reference design uses Temporal. A deterministic workflow owns state, timers, retry policies, signals and the compensation stack. Side-effecting work runs in activities with stable idempotency keys derived from (workflow_id, step_id). The workflow calls Claude for a bounded judgement task, receives structured output, and then decides how the process advances. The model never receives a tool that can start, signal or terminate the durable workflow itself.

PagerDuty / Datadog / ITSM events
               |
               v
      Durable IncidentWorkflow
   state · timers · retries · sagas
       /        |          \
      v         v           v
context pack   bounded      execute step
(read-only)    agent plan   activity
      |         |           |
      +---------+-----------+
                |
        evidence/effect log
                |
          human approval
                |
       customer / ITSM effects

This inversion is the main architecture decision: the workflow remembers; the model judges. A proposal activity may get eight to fourteen turns. It does not own an unbounded 200-turn incident lifecycle.

Compensation follows naturally. Each runbook step declares whether it is compensable and, if so, names the compensating step. If a later step fails, the workflow unwinds the recorded stack in reverse. Non-compensable effects—an email, a propagated DNS change, a schema migration—are explicit points of no return and should require a stronger human confirmation before crossing them.

If Temporal is not part of the target stack, the handbook describes a smaller alternative: a Postgres workflow state table plus an effect table with a unique idempotency key. That can still move the record of what happened outside the agent process. It is less feature-rich, but it preserves the essential property: restarting the model cannot erase or duplicate completed effects.

Web-edition qualification. Temporal’s current documentation describes durable, crash-resilient execution with replayable workflow state, signals, timers and retryable activities. Its Python SDK exposes execute_activity, wait_condition, workflow signals and retry policies. Treat the code below as a reference excerpt with application dependencies omitted; validate determinism, activity semantics and versioning against the installed SDK before deployment.

3. Repository layout and CLAUDE.md

operations-agent/
  CLAUDE.md
  .mcp.json
  .claude/
    agents/
      incident-context-assembler.md
      runbook-driver.md
      conformance-analyst.md
      ops-review-narrator.md
    skills/
      incident-comms-draft/SKILL.md
      weekly-operating-review/{SKILL.md,reference.md}
      runbook-execution-protocol/SKILL.md
    hooks/
      hooks.json
      ops-policy-gate.py
      freeze-calendar.py
  workflows/
    incident.py
    sla_watch.py
    recurring.py
    activities/{context.py,runbook.py,comms.py}
  runbooks/
    rb-0142-payment-gateway-degraded.yaml
  process_models/
    incident-management.pnml
  mcp/
    ops/server.py
    runbook/server.py
    pm/server.py
  evals/
    golden/{context_packs.jsonl,runbook_steps.jsonl,conformance.jsonl}
    chaos/injections.yaml
    rubrics/handoff_quality.md

Runbooks live as structured, reviewable YAML in version control, not only as prose in a wiki. A step that is an addressable record can carry its blast radius, dry-run behaviour, post-condition, compensation and verification logic. A wiki can remain the rendered view, but it should not be the executable source of truth.

CLAUDE.md carries invariants that are true on every run and expensive to violate. Long runbook catalogues, process models and account-level SLA matrices stay outside it.

# Operations Agent — operating invariants

## Absolute
- Never set an incident to Resolved, Closed or Cancelled.
- Never publish Sev1 customer communication without a named human approver.
- If tool state contradicts your recorded state, escalate_and_stop.

## Time
- Emit UTC ISO-8601 timestamps with explicit offsets.
- Refuse naive timestamps; do not guess a timezone.
- Order incident timelines by event time, not ingestion time.

## Severity
- Human authority sets severity; the agent may propose a change with evidence.

## Clocks
- The customer SLA outranks an internal OLA.
- Pause a clock only when the authoritative record says it is paused.

## Blast radius
- 0 read-only
- 1 reversible, single non-production CI
- 2 reversible, production, single CI
- 3 multiple CIs or customer-visible
- 4 irreversible or data-affecting
- Radius 3 requires two distinct human approvers.
- Radius 4 is never agent authority.

The exact severity taxonomy, account calendars and blast-radius definitions must be owned by operations/change management and versioned as policy. The source values are a reference model. A deployment should not hide organisation-specific change rules inside the prompt merely because the prose is convenient.

4. Subagents: narrow context, narrow tools

The chapter uses four roles. None both plans and executes. That separation is enforced through tool availability and backend identity, not only prompt text.

incident-context-assembler

---
name: incident-context-assembler
description: Assemble a bounded, deduplicated context pack for an active incident.
tools: mcp__ops__assemble_incident_context, mcp__itsm__search_changes, mcp__corpus__search
model: haiku
effort: low
maxTurns: 8
disallowedTools: Bash, Write, Edit, WebFetch
permissionMode: default
memory: project
background: false
color: cyan
---
Call assemble_incident_context exactly once.
Report truncated_sections verbatim.
Return, in order:
1. blast radius now
2. recent changes
3. alert signatures ordered by first_seen
4. at most five similar incidents, each with its resolution
5. open and related incidents/problems
Do not diagnose. If the context tool fails twice, escalate_and_stop.

Boundedness is part of correctness. A user must know when a section was truncated. Filling the missing area with a model summary defeats the whole character-budget design.

runbook-driver

---
name: runbook-driver
description: Drive one matched runbook step at a time against a live incident.
tools: mcp__runbook__get_runbook, mcp__runbook__execute_runbook_step, mcp__itsm__add_work_note, mcp__ops__escalate_and_stop
model: sonnet
effort: medium
maxTurns: 14
disallowedTools: Bash, Write, Edit, WebFetch, mcp__statuspage__publish_update
permissionMode: default
skills: runbook-execution-protocol
memory: project
color: orange
---
For each step:
1. fetch it and restate blast_radius and compensable
2. dry-run first; report proposed commands verbatim
3. obtain approval for this step, not a blanket prior approval
4. execute using the workflow-supplied idempotency key
5. check the post-condition
If verification fails, do not retry or continue: escalate_and_stop.
Write a work note after every executed step; the work note is the timeline.

The model never generates its own idempotency key. Stability across retries and replays is a workflow property; letting the model invent the key destroys it.

conformance-analyst

The conformance analyst does not decide whether a case conformed. A deterministic alignment tool compares the event log with the process model and returns fitness plus deviations—moves present in the log but not allowed at that point, and model-required moves that did not occur. The model then reads only the local evidence around those deviations and classifies likely causes such as control_gap, tooling, judgement, data_quality or training.

This is a useful pattern beyond ITSM: let deterministic process mining find where to look; use the language model to explain why the deviation happened and whether it mattered. The process model is authoritative for “should”; the event log is authoritative for “did.”

ops-review-narrator

The weekly narrator receives a governed operating pack and explains material variances. It can use a higher-reasoning model because the task is sparse and review-heavy, but it has no execution path. It must attach named cases and a mechanism to a variance or label it “not explained.” Avoid granting a background narrator operational tools simply because it can see the same metrics as the incident agent.

Web-edition subagent qualification. Current Claude Code documentation gives subagents their own context and tool restrictions. Plugin-defined subagents have different support for hooks, MCP configuration and permission modes than project subagents. Validate where each definition lives before assuming every frontmatter field in this reference repository is enforced the same way.

5. Skills: encode the procedure, not the current state

incident-comms-draft

---
name: incident-comms-draft
description: Draft incident communications from the live incident record.
allowed-tools: mcp__itsm__get_incident, mcp__ops__assemble_incident_context, mcp__corpus__search
disallowed-tools: mcp__statuspage__publish_update
user-invocable: true
arguments: [incident_id, audience]
---
# Incident communications draft
Audience is internal, account or public.

1. What is happening — observable impact in the customer's terms.
2. Who is affected — service scope, or say scope is not established.
3. What is being done — present tense; no internal CI names for public.
4. Next update — absolute UTC time from the contracted obligation.

Do not attribute cause before the postmortem.
Do not infer an ETA from a similar incident.
For Sev1, leave approved_by empty for a named human.

The current freeze posture and notification obligations should be injected from deterministic wrappers or tools at invocation time. They are live state, not prompt facts. A “next update” timestamp is a contractual commitment; it comes from the account obligation, not the model’s guess about what sounds reasonable.

The weekly operating-review skill uses the same progressive-disclosure pattern. Keep the main skill small: section order, variance materiality rules, the requirement to trace an explanation to named cases, and the explicit option to write “not explained.” Put metric definitions, account thresholds and worked prose examples in a companion reference loaded only when that section is needed.

Runbook execution is also a skill because it is a repeatable protocol. It should teach step-at-a-time behaviour, dry-run/approval/verification ordering and escalation rules. The runbook’s actual executable facts belong in the structured runbook record, so a skill release cannot silently change a production command.

6. Integrations and MCP contracts

SystemReference integrationEngineering constraint
ServiceNow / CMDBGateway-wrapped REST and shaped server-side endpoints.Use the actual service identity’s ACLs; keep write scope narrow to work notes/proposals.
Jira Service ManagementIssue search, comments and governed transitions.Do not treat a generic transition endpoint as permission to close authoritative incident state.
PagerDuty / OpsgenieIncident/log reads, notes and governed escalation.Normalise timestamps at the adapter boundary.
Datadog / GrafanaMetrics/events/monitor state.Bound queries and preserve source time and aggregation.
Slack / TeamsOperator UI, incident channel and approval surface.Mirror decisions/effects to durable evidence; chat retention is not the incident ledger.
Status page / communicationsPublication endpoint behind human approval.Sev1 publication is never an unconstrained agent tool.
Runbook storeGit-backed structured YAML.Read steps by immutable version; execute through a guarded adapter.
TemporalDurable workflow host.Not exposed to the model as MCP; application code calls the model.

Vendor rate limits in the handbook are reference observations, not current contractual guarantees. Slack, for example, documents tier-specific limits and returns Retry-After on HTTP 429; chat.postMessage is generally limited around one message per second per channel. The Events API expects fast acknowledgement and retries failed deliveries. A production adapter should queue work, respect current headers and reconcile delivery, rather than bake a table value into the prompt.

ServiceNow’s REST APIs apply ACLs and role checks for the authenticated identity. This is exactly why tests under an engineer’s account are insufficient evidence for an agent service principal. For context packs that need several joins, use a bounded server-side shape where appropriate rather than dumping multiple unbounded table reads into the transcript.

Context assembly owns its own budget

BUDGET = {
  "affected_ci_graph": 1500,
  "recent_changes": 3000,
  "alert_history": 2500,
  "similar_incidents": 4000,
  "open_related": 1000,
}

def collapse_alerts(alerts):
    signatures = {}
    for alert in alerts:
        key = (alert.monitor_id, alert.ci, alert.transition)
        row = signatures.setdefault(key, {"signature": alert.title, "count": 0,
            "first_seen": alert.ts, "last_seen": alert.ts})
        row["count"] += 1
        row["first_seen"] = min(row["first_seen"], alert.ts)
        row["last_seen"] = max(row["last_seen"], alert.ts)
    return sorted(signatures.values(), key=lambda r: r["first_seen"])

The tool collapses an alert storm before it reaches the context window and reports dropped rows per section. A generic maximum-result-size hint is a backstop. Section budgets and deterministic deduplication are the actual design.

Execution defaults to dry-run in the handler

async def execute_runbook_step(args):
    # Reference adapter excerpt; dependencies are application services.
    dry_run = bool(args.get("dry_run", True))
    step = await runbooks.step(args["runbook_id"], args["step_id"])
    missing = [ci for ci in step.cis if not await cmdb.exists(ci)]
    if missing:
        return err(f"runbook stale; missing CIs: {missing}")
    if int(args.get("blast_radius", -1)) != step.blast_radius:
        return err("declared blast radius differs from the runbook")
    if not dry_run:
        approval = await approvals.verify(args.get("approval_token"), step)
        if not approval.valid:
            return err(approval.reason)
        if step.blast_radius >= 3 and len({s.principal for s in approval.signers}) < 2:
            return err("two distinct approvers required")
    prior = await effects.get(args["idempotency_key"])
    if prior:
        return ok({**prior.result, "replayed": True})
    result = await executor.run(step, dry_run=dry_run)
    if not dry_run:
        await effects.put(args["idempotency_key"], result)
    return ok(result)

The important property is not the exact Python. Safety defaults live in application code, approvals are verified against authenticated records, and the effect log is checked before execution. A JSON schema default or prompt instruction is not enough.

7. Retrieval: incidents are noisy, runbooks decay, process is an event log

The corpus contains runbooks, incidents and postmortems, change records, architecture decision records, operating-model documentation and SLA/OLA schedules. It should not become a raw Slack archive or a credential graveyard. Customer-sensitive names and postmortem content remain behind entitlement filters enforced in the retrieval service.

Chunking follows the object being reasoned about. Runbooks are chunked by executable step, because the step is the unit of dry-run, approval, execution and verification. Postmortems are chunked by section. Change records keep structured fields as fields and embed only meaningful implementation/backout text.

Similar incidents: remove the template before embedding

Ticket text is dominated by boilerplate—form headers, integration-user lines, alert payloads and repeated status text. Embedding it makes “similar” mean “uses the same ticket template.” The reference design first strips stable template blocks, then classifies remaining content into symptom narrative, diagnosis, resolution and boilerplate. It embeds symptom and resolution separately.

Before vector search, filter candidates using structured signals such as service, affected-CI class, controlled symptom type and explicit error code. Then query the symptom index to answer “have we seen this?” and the resolution index to answer “what fixed cases that look like this?” Fuse, rerank and return a small evidence set. A similar incident without its resolution is trivia, not guidance.

Runbook retrieval is health-aware

A runbook can be semantically relevant and operationally dangerous because the environment changed around it. Maintain a health record using signals such as last_verified_at, CMDB drift and recent execution failure rate. Penalise stale runbooks; exclude them entirely once they cross a reviewed staleness limit, and route them to a decay report for their owner.

The source’s 0.80 exclusion line, 31% unverified-in-twelve-months observation and 9% decommissioned-CI observation are reference-deployment figures. They illustrate why staleness belongs in retrieval ranking; they are not thresholds or performance claims for a new installation.

Process mining is not prose retrieval

The event log is ground truth for what happened. A model asked to infer process from documentation gives you the documented process—the gap between documented and actual behaviour is exactly what conformance analysis needs. Use deterministic optimal alignment between the event sequence and the process model, then pass the resulting deviation windows to the model for explanation.

PM4Py’s current documentation provides Petri-net alignment algorithms including A*-style/state-equation and Dijkstra variants. The choice of model, cost function and event normalization is part of the governed process-mining implementation. The LLM should not silently invent alignment costs or a process model because the library can calculate them.

8. Guardrails: stop must be a first-class outcome

The operations policy gate protects four classes of boundary: authoritative incident state, customer communication, production execution and change freezes. Backend identity and tool absence remain the primary controls; hooks add a fast, auditable policy layer.

# Reference policy logic, simplified
if tool == "update_incident" and state in {"resolved", "closed", "cancelled"}:
    hard_block("a named human owns terminal incident state")
if customer_facing and severity == 1 and not authenticated_approver:
    hard_block("Sev1 customer communication requires a named human approver")
if executing_runbook_step:
    if blast_radius > context_ceiling:
        deny("blast radius exceeds this context's ceiling")
    if change_freeze_active and not authorized_emergency_change:
        hard_block("active freeze requires an authorised emergency change")
    if blast_radius >= 3 and distinct_approvers < 2:
        hard_block("two-person rule")

Web-edition hook correction. The handbook comments that a JSON deny may be overridden by a later allow. Current Claude Code hook documentation says matching PreToolUse decisions are combined using the most restrictive result: deny > ask > allow. Exit code 2 still blocks the tool call. Treat the current documented semantics as authoritative and test the installed release; do not carry the older “later allow overrides deny” comment into production.

A hook is not the whole authorization system. The service principal must lack terminal-state or broad production credentials. An approved step carries the human identity through the gateway, and the target system enforces that human’s entitlements. Someone who cannot restart a service directly cannot grant an agent more power through a chat card.

escalate_and_stop

The less obvious guardrail is a tool that lets the agent admit it has lost the state. Triggers include repeated idempotency conflicts, a tool result contradicting recorded workflow state, failed compensation, or a context pack whose truncation hides the CI under investigation. The tool writes a handoff pack, notifies the incident commander and moves the durable workflow into an explicit human-handoff state.

The handoff pack says what was executed, what definitely landed, what may be half-done, what evidence conflicts, and the single next thing the human should check. “Stop cleanly” is a production capability. An agent without it tends to improvise precisely when the environment is least predictable.

9. Production deployment: workflow state is durable, agent state is disposable

The following reference excerpt preserves the handbook’s topology while making the separation explicit. Activity implementations, approval validation, compensation logic and the Claude adapter are application code and are omitted here.

from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy

@workflow.defn
class IncidentWorkflow:
    def __init__(self):
        self.approved = {}
        self.compensations = []

    @workflow.signal
    def approve(self, step_id: str, token: str):
        self.approved[step_id] = token

    @workflow.run
    async def run(self, event: dict):
        pack = await workflow.execute_activity(
            assemble_context, event["incident_id"],
            start_to_close_timeout=timedelta(minutes=4),
            retry_policy=RetryPolicy(maximum_attempts=3))
        plan = await workflow.execute_activity(
            propose_plan, {"pack": pack, "incident_id": event["incident_id"]},
            start_to_close_timeout=timedelta(minutes=6),
            retry_policy=RetryPolicy(maximum_attempts=2))
        for step in plan["steps"]:
            sid = step["step_id"]
            if step["blast_radius"] >= 1:
                await workflow.execute_activity(
                    request_approval, step,
                    start_to_close_timeout=timedelta(minutes=1))
                await workflow.wait_condition(
                    lambda: sid in self.approved,
                    timeout=timedelta(minutes=20))
            stable_key = f"{workflow.info().workflow_id}:{sid}"
            result = await workflow.execute_activity(
                run_step,
                {"step": step, "idempotency_key": stable_key,
                 "approval_token": self.approved.get(sid)},
                start_to_close_timeout=timedelta(minutes=10),
                retry_policy=RetryPolicy(maximum_attempts=4))
            if result.get("compensable"):
                self.compensations.append((sid, result["compensating_step"]))

Temporal workflow code must remain deterministic under replay. Do not generate a random idempotency key inside the activity boundary and then expect replay to recover it. Persist or derive keys from durable workflow identity and step identity. Treat changes to timer/condition behavior as versioned workflow changes where required by the SDK’s determinism rules.

The Claude activity is deliberately bounded

async def propose_plan(input_data):
    # Reference excerpt. Host configures actual SDK types and structured parsing.
    options = {
        "model": "approved-worker-model",
        "allowed_tools": ["mcp__runbook__get_runbook", "mcp__corpus__search",
                          "mcp__ops__escalate_and_stop"],
        "disallowed_tools": ["mcp__runbook__execute_runbook_step",
                             "Bash", "Write", "Edit", "WebFetch"],
        "max_turns": 12,
    }
    return await call_agent_for_structured_plan(input_data, options)

The planner cannot execute. Execution lives in another activity with another capability set. This is stronger than asking the same session to “be careful” after it has persuaded itself that urgency justifies a shortcut.

Retries live on activities with explicit idempotency contracts. A timeout after a possibly completed write is not automatically safe to repeat; authoritative readback or the effect log determines whether the effect exists. Budget/turn caps are circuit breakers: if the bounded planner cannot converge, the correct state is human handoff, not a larger autonomous loop.

Verification boundary. The reference Python snippets above are structurally consistent with the current Temporal Python workflow APIs and are presented as excerpts, not as a runnable repository. Production requires workflow/activity registration, data converters, exception typing, compensation implementation, SDK version pinning and tests against a sandbox target system.

10. Evaluation: chaos-test the control plane

Operations evaluation cannot stop at answer quality. The core claim is that workflows survive crashes without duplicate external effects and that humans receive a usable handoff when the agent stops. The reference evaluation set therefore includes context packs, runbook-step labels, process-conformance cases and mid-incident handoff snapshots.

MetricWhat it testsHandbook reference gate
Causal-change recall@3The known causal change appears among the top three changes in the context pack.At least 0.85; hard fail.
Pack-truncation honestyEvery non-empty truncated_sections is surfaced.1.000; hard fail.
Runbook-step accuracyProposed next step matches senior-engineer labels.At least 0.90; hard fail.
Stale-runbook selectionSelected runbooks above the reference staleness band.No more than 0.02; hard fail.
Conformance kappaDeviation-cause classification vs process analyst.At least 0.75; warning.
Handoff qualityExecuted state, confirmed-vs-assumed, next check and timestamp clarity.At least 2.5/3; hard fail.
Duplicate-effect rateMore than one external effect for one idempotency key under injected failures.0.000; hard fail.
Resume-success rateWorkflow completes after worker kill/restart.At least 0.99; hard fail.

These values describe the handbook’s reference programme. A new deployment should derive acceptance thresholds from its own incident severity, contractual obligations and failure costs. What should remain invariant is the shape of the test: a side-effecting orchestration agent needs chaos and recovery evaluation, not only a prompt-quality judge.

Kill it at the awkward point

The chaos suite runs against sandbox ITSM/PagerDuty-style endpoints, kills the worker at an activity boundary, during an activity before the effect, and after the effect but before the activity result is recorded. Restart it and assert: the workflow completes, each idempotency key produces at most one effect, and no compensation remains dangling.

The source reports that this style of test found a random UUID generated inside the wrong boundary, a non-deduplicated Slack post and a non-idempotent compensation. Those are reference failure observations, not Infinity client incidents. Recreate the failure classes in your own sandbox and make the gate deterministic.

MTTA and MTTR belong in operational trend analysis, not a CI prompt gate. They are confounded by incident mix, staffing and changing classification. Compare service lines and longer windows; do not claim the model improved MTTR because one quarter moved after deployment.

11. Failure modes and their controls

Duplicate side effects after retry

A status page or work-note update lands, the activity crashes before recording success, and the retry emits it again. Use a stable idempotency key, check the effect log before execution and record the result under a unique constraint. Exactly-once business effects are a property of the adapter plus store, not a property of a model transcript.

A stale runbook executes confidently

The step text is perfect but names a CI retired fourteen months ago. Penalise stale runbooks during retrieval and assert that every named CI resolves in the CMDB immediately before execution. Do not “helpfully” map an old hostname to its replacement inside the model.

Alert-storm context flooding

Thousands of downstream alerts cause the model to anchor on the loudest symptom. Collapse alerts into signatures inside the context tool and order by first-seen evidence rather than count. The handbook’s 3,400-alert example collapsing to six signatures demonstrates the pattern; it is an illustrative reference observation.

The timeline is reconstructed in the wrong timezone

One system renders local instance time, another emits UTC, and the postmortem reverses causal order by minutes. Normalize at the MCP boundary, require timezone-aware timestamps and carry a tz_normalised marker. Refuse a naive timestamp rather than guessing the zone.

Local OLA optimization breaks the customer SLA

A routing action can look beneficial to one queue while consuming the global contractual clock. Put the objective hierarchy in the operating invariants and make routing tools return the effect on customer clocks, not only local backlog.

The organisational failure: the agent subsidises preventable work

If the agent becomes cheap and competent at recurring defects, teams can stop fixing the defect. Classify closed work into genuine incident, known-defect-not-fixed, process gap, should-be-self-service and data-quality categories, attach a cost, and report the avoidable categories to the service owner. Automation should reduce the source of operational demand, not merely make the queue more tolerable.

All failure examples in this guide come from the supplied reference handbook or from explicitly labelled web-edition hardening. They are engineering test cases, not claims about incidents or measured outcomes at Infinity customers.

12. Build order: earn write authority only after recovery works

Week 1: stand up durable workers, the effect log with a unique idempotency constraint, read-only context assembly across ITSM/PagerDuty/observability, and the evidence timeline. Execute nothing. Measure whether the bounded pack is useful enough that on-call engineers stop opening the same tabs.

Month 1: convert runbooks to step-level structured YAML in git, add staleness scoring and the first decay report, keep runbook-driver dry-run-only, introduce L1 communications drafting, and run conformance checking on one documented process. Build the chaos suite before enabling a live runbook step.

Quarter 1: enable carefully selected blast-radius-1 execution at L2 with step-scoped approval and the two-person rule wired for larger actions, add CAB impact assessment, the weekly operating-review pack and “tickets that should not exist” reporting. The handbook explicitly keeps L3 off until the chaos suite is clean across two releases.

The permanent architecture rule remains: the workflow engine owns state, the model owns bounded judgement, deterministic tools own measurements and alignments, and humans own consequential state transitions. Increasing model capability does not change those responsibilities.

Primary sources and web-edition notes

The structural source is Article 7 of the supplied Enterprise Agent Mesh handbook, with the shared platform from Part 0. The links below support the current implementation qualifications. They do not turn the reference excerpts into a complete vendor integration.

Continue the Enterprise Agent Mesh series

← Previous: 06 Finance Risk Agent — Evidence Assembly, Adversarial Review, and Model Risk Management

01 Recruitment Agent · 02 HR Agent · 03 Supply Chain Agent · 04 Procurement Agent · 05 Finance Agent

Next: 08 Customer Support Agent — Containment Quality, Not Deflection Rate. Forthcoming.

Infinity Technologies
Enterprise Agent Mesh Engineering Guides
September 2026
No items found.

Our success stories

Enterprise Agent Mesh platform illustration for the Compliance Agent guide, with blue hexagonal agents and a central governance shield.
September 2026

Compliance Agent: Evidence Logistics, Control Testing, and the Mesh’s Control Plane

Engineering guide 12/12: build a Compliance Agent for reproducible control testing, sealed evidence, curated crosswalks and continuous mesh conformance.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
AI/ML
Shared Enterprise Agent Mesh illustration for the Legal Agent engineering guide.
September 2026

Legal Agent: The Playbook Is the Program

Engineering guide 11/12: build a Legal Agent where executable playbooks, matter access, contract lineage and privilege boundaries govern model-assisted review.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
AI/ML
Shared Enterprise Agent Mesh illustration for the Marketing Agent engineering guide.
September 2026

Marketing Agent: Generation Is the Commodity, the Constraint System Is the Product

Engineering guide 10/12: make generation subordinate to market-scoped claims, rights, channel rules, attribution discipline and measurable brand controls.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
AI/ML
Shared Enterprise Agent Mesh illustration for the Sales Agent engineering guide.
September 2026

Sales Agent: Make the CRM True, Then Worry About Selling

Engineering guide 09/12: make CRM state evidence-backed before generating selling assistance, with governed field updates, customer commitments and cross-agent handoffs.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
AI/ML
Shared Enterprise Agent Mesh illustration for the Customer Support Agent engineering guide.
September 2026

Customer Support Agent: Containment Quality, Not Deflection Rate

Engineering guide 08/12: containment quality, account-scoped retrieval, guarded customer sends, escalation packets and evidence-driven support autonomy.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
AI/ML
Shared Enterprise Agent Mesh illustration for the Operations Agent engineering guide.
September 2026

Operations Agent: Durable Execution, Process Conformance, and the Connective Tissue of the Mesh

Engineering guide 07/12: durable workflow state, bounded judgement, process conformance, runbook safety and chaos-tested recovery.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
AI/ML
Shared agent-platform illustration for the Finance Risk engineering guide: connected hexagons surrounding a governance core.
September 2026

Finance Risk Agent: Evidence Assembly, Adversarial Review, and Model Risk Management

Engineering guide 06/12: deterministic treasury calculations, facility-specific covenant definitions, adversarial challenge and human decision authority.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
AI/ML
Enterprise Agent Mesh platform illustration for the Finance Agent engineering guide.
September 2026

Finance Agent: Numbers From Tools, Never From the Model

A production engineering guide to a Finance AI Agent where every figure comes from deterministic tools and immutable fact packs, while the model is limited to grounded narrative and workflow orchestration.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
AI/ML
Governed enterprise agent platform with specialized nodes around a protected control core.
September 2026

Procurement Agent: Segregation of Duties Encoded in the Tool Layer

Engineering guide 04/12: build a Procurement Agent where approvals, supplier banking and payment authority are structurally outside the model’s tool and credential boundary.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
AI/ML
Shared enterprise agent platform illustration used for the Supply Chain Agent engineering guide.
September 2026

Supply Chain Agent: Exception Narratives Over an Optimiser You Already Own

Engineering guide 03/12: build a Supply Chain Agent that triages planning exceptions, explains shortage causality with provenance and delegates quantities to deterministic solvers.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
AI/ML
Governed enterprise agent platform connecting specialist agents around a protected control core.
September 2026

HR Agent: Effective-Dated, Jurisdiction-Scoped Policy Retrieval

Engineering guide 02/12: build an HR Agent that resolves employee context before retrieval, answers against effective-dated policy and routes sensitive cases.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
AI/ML
Governed enterprise agent mesh: connected hexagonal agents around a protected platform core.
September 2026

Recruitment Agent: Evidence Assembly Under a High-Risk Regulatory Regime

Engineering guide 01/12: build a Recruitment Agent that assembles requirement-linked evidence, preserves provenance and keeps candidate decisions with humans.
Infinity Technologies
Enterprise Agent Mesh Engineering Guides
AI/ML
September 2026

AI Model Router Agent — Engineering Guide

Engineering guide 12/12: route tasks across Claude, Codex and self-hosted models using data policy, capabilities, evaluation scores, cost and availability.
Infinity Technologies
InfinitySDLC Engineering Guides
AI/ML
September 2026

Incident Response Agent — Engineering Guide

Engineering guide 11/12: build an incident copilot with structured state, specialist-agent handoffs, typed runbooks and human-approved mitigation.
Infinity Technologies
InfinitySDLC Engineering Guides
AI/ML
September 2026

Risk & Reliability Agent — Engineering Guide

Engineering guide 10/12: quantify change risk using SLOs, error budgets, dependency graphs and resilience evidence rather than an ungrounded model score.
Infinity Technologies
InfinitySDLC Engineering Guides
AI/ML
September 2026

Threat Detection Agent — Engineering Guide

Engineering guide 09/12: enrich SIEM and EDR alerts, resolve entities and build evidence-backed incident timelines without unbounded containment powers.
Infinity Technologies
InfinitySDLC Engineering Guides
AI/ML
September 2026

Security Prevention Agent — Engineering Guide

Engineering guide 08/12: combine deterministic security scanners, threat-model RAG and contextual code review before merging software changes.
Infinity Technologies
InfinitySDLC Engineering Guides
AI/ML
September 2026

Observability Agent — Engineering Guide

Engineering guide 07/12: correlate traces, metrics, logs and deployments using bounded telemetry queries and evidence-backed competing hypotheses.
Infinity Technologies
InfinitySDLC Engineering Guides
AI/ML
September 2026

Change & Release Orchestration Agent — Engineering Guide

Engineering guide 06/12: coordinate change approval, CI/CD, progressive delivery and rollback with deterministic state transitions and two-phase writes.
Infinity Technologies
InfinitySDLC Engineering Guides
AI/ML
September 2026

QA & Validation Agent — Engineering Guide

Engineering guide 05/12: build a QA agent that selects risk-based tests, uses isolated coding agents and produces verifiable release evidence.
Infinity Technologies
InfinitySDLC Engineering Guides
AI/ML
September 2026

Environment Agent — Engineering Guide

Engineering guide 04/12: build an Environment Agent for reproducible infrastructure, bounded Kubernetes diagnostics and disposable test environments.
Infinity Technologies
InfinitySDLC Engineering Guides
AI/ML
September 2026

Planning & Architecture Agent — Engineering Guide

Engineering guide 03/12: convert approved requirements into architecture decisions, dependency-aware delivery plans and machine-checkable work packages.
Infinity Technologies
InfinitySDLC Engineering Guides
AI/ML
September 2026

Product Discovery Agent — Engineering Guide

Engineering guide 02/12: turn customer feedback, product analytics and repository context into evidence-backed hypotheses and traceable requirements.
Infinity Technologies
InfinitySDLC Engineering Guides
AI/ML
September 2026

Enterprise Agent Platform Foundation — Engineering Guide

Engineering guide 01/12: build the shared control plane, MCP gateway, ACL-aware retrieval, isolated runtimes and evaluation system for an enterprise AI agent mesh.
Infinity Technologies
InfinitySDLC Engineering Guides
AI/ML

Optimized Warehouse Process Saves 4 Million USD per Annum for a Nation-Wide Logistics Operator

AI-powered warehouse monitoring system with real-time stress detection and worker coordination
AI/ML
Web Development

Infinity Technologies for HORSCH

Automated tracking for 4,000+ supplier components
AI/ML
CRM/ERP
IoT
Mobile Development
Web Development

ATLAS Manager CoPilot

AI Co-Pilot helps telecom managers cut admin work and lead teams.
AI/ML
CRM/ERP
Mobile Development
Web Development

AI Recruiting Assistant

AI assistant inside SAP cuts hiring time
AI/ML
CRM/ERP

Project ONIX: AI-Driven Employee Onboarding for a Fortune 200 Oil & Gas Operator

AI onboarding agent streamlines HR for an oil and gas firm
AI/ML

Project MERIDA: An Enterprise Knowledge Assistant for a European Bank

AI knowledge assistant for a European bank
AI/ML
CRM/ERP

AI Employee Knowledge Assistant

AI assistant for a European bank to centralize internal knowledge
AI/ML
CRM/ERP

Gemini Enterprise Churn Explanation and Retention

AI assistant for churn explanation and retention
AI/ML
CRM/ERP

Gemini Enterprise Call Center Assistant

AI assistant for telemedicine
AI/ML
CRM/ERP

Gemini Enterprise Identity Verification Assistant

AI assistant for KYC to reduce onboarding drop-offs
AI/ML
Mobile Development
Web Development

Gemini Enterprise Photo-Driven Router Setup Assistant

Multimodal AI on Gemini Enterprise lets customers photograph their router instead of describing it for guided self-setup
AI/ML

AI-Driven Network Planning and Capacity Expansion for Mobile and Fixed Telecom Networks

AI-driven network planning platform for telecom operators with predictive demand and capex optimization
AI/ML
IoT

AI-Driven Telecom Fraud Detection & Prevention Platform

AI-driven real-time fraud detection and prevention platform for telecom networks
AI/ML
CRM/ERP

AI-Driven Predictive Field Maintenance for Towers & RAN Equipment

AI-driven predictive maintenance and field operations optimization platform for telecom network infrastructure
AI/ML
IoT

AI-Driven Next Best Action (NBA) Engine for BSS

AI-driven revenue assurance platform for telecom BSS with real-time anomaly detection and automated correction
AI/ML
CRM/ERP

NetAssure AI — Autonomous Service Assurance for RAN/Core

AI-powered closed-loop network operations platform for telecom service assurance and energy optimization
AI/ML
CRM/ERP
IoT

AI-Driven Predictive Maintenance for Rotating Equipment


AI predictive maintenance platform for rotating equipment at a gas processing facility
AI/ML
IoT
Web Development

“Explainable line” copilot (LLM over event log + manuals)

Explainable AI copilot that turns PLC logs and manuals into clear explanations and troubleshooting guidance
AI/ML
IoT
Web Development

Operator behavior & training insights

Operator coaching and best-practice analytics using HMI/PLC interaction data to stabilize performance across shifts
AI/ML
IoT
Web Development

Safety & Near-Miss Analytics for Industrial Production Lines

Safety & near-miss analytics system using PLC safety signals and AI scenario detection for industrial production lines.
AI/ML
IoT
Web Development

Automatic parameter recommendation (“recipe optimization”)

AI-based recipe optimization system for automatic tuning of temperatures, speeds, and pressures on production line.
AI/ML
IoT
Web Development

Quality Analytics: Veneer Thickness, Cut Quality & Defects

AI-driven veneer thickness and cutting quality analytics for production line
AI/ML
IoT
Web Development

Buffer & Bottleneck Optimization Across the Production Line, Storage, and Lift System

Digital twin and AI optimization for buffer flow and bottleneck management on production line
AI/ML
IoT
Web Development

Predictive maintenance of drives & motion axes

Predictive maintenance solution for drives and motion axes on production line.
AI/ML
IoT
Web Development

Predictive maintenance for heating & glue system

Predictive maintenance solution for the Heating & Glue system on production line.
AI/ML
IoT
Web Development

Full OEE and Lost‑Hours Analytics for Production Line

Digital OEE and lost-hours analytics solution for high-throughput veneer production line
AI/ML
IoT
Web Development

Automatic stop detection & classification: Micro-stop Analytics

Advanced micro-stop analytics with ML-assisted classification and root-cause insights
AI/ML
IoT
Web Development
CRM/ERP

Automatic stop detection & classification: Microstop

Micro-stop Monitor detects and classifies short production stops using PLC data and rules
AI/ML
IoT
Web Development

AI Regulatory & Licensing Compliance Copilot

AI copilot for regulatory and licensing compliance across multiple jurisdictions
AI/ML
CRM/ERP
Web Development

AI Due Diligence Platform for M&A and New Projects

AI platform for automated M&A and new project due diligence
AI/ML
CRM/ERP
Web Development

Autonomous Dispatch & BESS AI Optimization

AI engine for portfolio dispatch and BESS optimization across volatile energy markets
AI/ML
IoT
Web Development

AI Asset Health & Degradation Prediction System

AI system for asset health monitoring and degradation prediction
AI/ML
IoT
Web Development

AI CO ₂ Calculator & ESG Impact Platform

AI-powered platform for automated CO₂ accounting and ESG reporting
AI/ML
Web Development

Cross-recipe: Energy vs Quality Analysis

A data-driven system optimized veneer press energy usage while maintaining product quality.
AI/ML
IoT
Feedforward Press Correction

Feedforward Press Correction

A leading engineered wood manufacturer implemented a predictive press control system powered by data and machine learning.
AI/ML
IoT
Web Development
“Bad-Sheet” Routing

“Bad-Sheet” Routing

Automated system for detecting and routing defective veneer sheets using real-time sensor data and analytics.
AI/ML
IoT

Early Fan Failure Detection

Plant A deployed an on-prem predictive maintenance system for fans, reducing unplanned downtime by 38%.
AI/ML
Predictive Hydraulic Filter Change

Predictive Hydraulic Filter Change

Predictive maintenance system for hydraulic filters reduced downtime and optimized maintenance scheduling in a large industrial plant.
IoT
AI/ML

Infinity Technologies in PetTech

A smart genetic testing platform that helps pet owners and breeders easily access and understand their pets’ DNA insights through a single digital solution.
AI/ML
IoT
Mobile Development

Intelligent Budgeting: How AI-Powered Financial Planning Transforms Business Strategy

A case study on how intelligent budgeting transformed financial planning, decision-making, and organizational agility.
AI/ML
CRM/ERP
Smarter Product Management Through Interactive Constructors and Real-Time Analytics

Smarter Product Management Through Interactive Constructors and Real-Time Analytics

An interactive, analytics-powered product constructor enabled smarter pricing, faster product decisions, and improved profitability across a complex portfolio.
AI/ML
CRM/ERP
The Power of Precision: How One Company Achieved 99.1% Sales Forecast Accuracy

The Power of Precision: How One Company Achieved 99.1% Sales Forecast Accuracy

A large-scale sales forecasting system achieved 99.1% accuracy across hundreds of products using data-driven, automated models.
AI/ML
CRM/ERP
Smarter Energy Forecasting in Manufacturing

Smarter Energy Forecasting in Manufacturing: Turning Data Into Cost Savings

A real-world case study on how predictive energy forecasting helps manufacturers cut costs and improve efficiency.
IoT
AI/ML
CRM/ERP
Smarter Hatching: How Predictive Modeling Transforms Poultry Incubation

Smarter Hatching: How Predictive Modeling Transforms Poultry Incubation

A poultry farm used AI and real-time data to optimize incubation, improving chick quality and operational efficiency.
AI/ML
IoT
Web Development
Predictive Analytics in Healthcare: The Future of Cardiovascular Risk Detection

Predictive Analytics in Healthcare: The Future of Cardiovascular Risk Detection

Predictive analytics model for early cardiovascular risk detection using non-invasive population data.
AI/ML
CRM/ERP
Smart Fraud Detection: How Predictive Analytics is Reshaping Social Welfare Systems

Smart Fraud Detection: How Predictive Analytics is Reshaping Social Welfare Systems

A public agency used predictive analytics to overhaul fraud detection in social welfare distribution.
AI/ML
Risk-Based Oversight of Social Benefits: Catching Fraud Without Hiring More Staff

Risk-Based Oversight of Social Benefits: Catching Fraud Without Hiring More Staff

Case study: shifting from random checks to risk-based fraud detection in social benefits.
AI/ML
Predicting Employee Turnover: How Data Turns Retention into a Strategy

Predicting Employee Turnover: How Data Turns Retention into a Strategy

This article explores how predictive analytics is transforming employee retention from a reactive process into a strategic advantage.
AI/ML
IoT
CRM/ERP
How Predicting Customer Churn Helps Banks Grow: A Case Study with 1500% ROI

How Predicting Customer Churn Helps Banks Grow: A Case Study with 1500% ROI

A real-world case study showing how predictive analytics helped a bank cut churn by 71% and achieve 1500% ROI through targeted retention.
AI/ML
CRM/ERP
Smarter Compliance: How Automated Risk Assessment Transforms Contractor Fraud Detection in Banking

Smarter Compliance: How Automated Risk Assessment Transforms Contractor Fraud Detection in Banking

This article explores how automated risk classification enhanced fraud detection and compliance efficiency in banking.
AI/ML
Smarter Loan Campaigns with Predictive Models

Smarter Loan Campaigns with Predictive Models

How predictive analytics helps banks improve cross-selling by reducing risk, cutting waste, and targeting the right customers.
CRM/ERP
AI/ML
Predictive Modeling Cuts Marketing Costs by 93% in Banking Campaign

Predictive Modeling Cuts Marketing Costs by 93% in Banking Campaign

A bank applied predictive modeling to identify high-response customers, reducing campaign costs from full budget to just 7% while maintaining results.
AI/ML
Risk-Based Personalization Boosts SME Overdraft Lending

Risk-Based Personalization Boosts SME Overdraft Lending

A major European bank revamped its SME overdraft lending by introducing a data-driven model that adjusted loan limits based on individual risk profiles, boosting both portfolio size and profit.
AI/ML
CRM/ERP
From 4 Months to 30 Minutes: The New Speed of Credit Scoring
August 2025

From 4 Months to 30 Minutes: The New Speed of Credit Scoring

A bank cut credit model time from four months to 30 minutes by automating risk assessment for corporate clients.
AI/ML
Nova Poshta: AI-Powered Warehouse Monitoring for Conveyor Systems

Nova Poshta: AI-Powered Warehouse Monitoring for Conveyor Systems

Infinity Technologies Builds Real-Time Load Balancing and Bottleneck Detection for Ukraine’s Largest Logistics Operator
AI/ML
CRM/ERP
IoT
Web Development