Observability Agent — Engineering Guide

InfinitySDLC Engineering Guides · 07/12

Reference implementation guide, not a report of a completed client deployment. Code, configurations, metrics and policies are illustrative. Adapt and validate them before production use.

The Observability Agent turns telemetry into a causal investigation workspace. It should correlate traces, metrics, logs, topology, deployments and ownership while avoiding the common failure mode of summarizing thousands of log lines without structure.

7.1 Prefer structured query tools over raw data dumps

• Metrics MCP: query_range(metric, filters, window), anomaly candidates, SLO burn rate.
• Trace MCP: search traces by service/span/error/tag; fetch bounded trace trees.
• Logs MCP: parameterized queries with field extraction and sampling.
• Topology MCP: service dependencies and owners.
• Change MCP: deployments, feature flags, config changes and incidents in the same time window.

Expose query templates and safe query languages. A model-generated PromQL/LogQL/SQL string should be validated, bounded by time range and result size, and executed with a read-only identity.

7.2 Correlation workflow

1. Detect or receive an anomaly.
2. Normalize the time window and affected SLO/service.
3. Fetch deployment/config/flag changes around the window.
4. Compare golden signals to a previous baseline.
5. Drill into exemplars/traces, then logs for the failing path.
6. Construct a causal graph with evidence strength, not a single confident story.
7. Link the finding to owner/runbook and hand off to Incident Response when intervention is required.

finding:
  symptom: checkout p95 increased 210ms -> 760ms
  started_at: 14:07Z
  correlated_change: deploy:pricing-api@7ad19f (14:03Z)
  trace_evidence: 71% of slow traces wait on pricing-api /quote
  metric_evidence: pricing-db pool saturation 96%
  confidence: 0.91
  next_safe_check: inspect connection-pool config diff

7.3 RAG and memory

Index runbooks, service docs, past incident postmortems and ownership metadata. Do not vectorize raw telemetry as the primary method; telemetry is temporal and should be queried live. Persist compact investigation summaries with source pointers so future sessions can resume without replaying gigabytes of logs.

7.4 Evaluation

• Time-to-first-useful-hypothesis on replayed incidents.
• Root-cause ranking: whether true cause appears in top-k hypotheses.
• Query efficiency: number of high-cost telemetry queries per investigation.
• Hallucination rate: claims not supported by a tool result or documented source.
• Prompt-injection resilience in log fields, commit messages and external payloads.

Implementation Blueprint: Query Broker + Causal Graph

Telemetry query broker

Do not give the model unlimited PromQL/LogQL/SQL. Expose typed intents that a broker compiles into backend queries, while enforcing time-window, cardinality, cost and result-size limits. Return both a compact result and a query ID so an engineer can reproduce it.

{
  "metric": "http.server.duration.p95", "service": "checkout",
  "window": "30m", "compare_to": "previous_day_same_window",
  "group_by": ["route"], "max_series": 50
}

Hypothesis object

Hypothesis { id, text, confidence,
  supporting_evidence:[{type,ref,strength}],
  contradicting_evidence:[{type,ref,strength}],
  next_discriminating_query, status }

Force the agent to maintain competing hypotheses until a discriminating query separates them. This limits “first correlated deployment = root cause” behavior. Replay real incidents and grade whether the true root cause appears in top-k hypotheses, not whether the narrative sounds convincing.

Source and Shared Prerequisites

Adapted from the September 2026 Enterprise AI Agent Mesh handbook, Article 7 and Blueprint 7. The Enterprise Agent Platform Foundation guide provides the shared identity, MCP, retrieval, sandbox, audit and evaluation design, plus the source handbook’s further-reading list. Validate model, protocol and tool versions before production use.

Explore the Series

Previous guide · Series foundation · Next guide