AI Model Router Agent — Engineering Guide

InfinitySDLC Engineering Guides · 12/12

Reference implementation guide, not a report of a completed client deployment. Code, configurations, metrics and policies are illustrative. Adapt and validate them before production use.

A hybrid enterprise agent stack needs an explicit model-selection layer. Without it, teams hard-code provider choices into prompts and lose control over privacy, cost, latency, capability and fallback behavior.

12.1 Routing dimensions

12.2 Policy-first routing

def route(task):
    candidates = registry.models_supporting(task.capabilities)
    candidates = [m for m in candidates if task.data_class <= m.max_data_class]
    candidates = [m for m in candidates if task.agency <= m.max_agency]
    scored = [
      (m, 0.45*eval_score(m, task.type) + 0.20*latency_score(m)
          + 0.20*cost_score(m) + 0.15*availability_score(m))
      for m in candidates
    ]
    return max(scored, key=lambda x: x[1])[0]

Policy filtering must happen before quality/cost scoring. If a task is restricted to on-prem inference, a superior cloud model is not a candidate. Model names should not appear in business prompts; agents request capability classes such as “frontier-code” or “local-reasoning,” and the router resolves the current model.

12.3 Suggested 2026 tiers

Treat these as examples, not permanent rankings. Re-run your own golden tasks whenever a model, quantization, runtime or prompt template changes. A 24B model specialized for code can beat a much larger general model on the exact workflows that matter.

12.4 Fallback and ensemble patterns

• Same-provider fallback for transient capacity issues.
• Cross-provider fallback only when data policy permits.
• Local fallback for continuity during external provider outage.
• Escalation: compact local model handles easy tasks and escalates ambiguous/high-risk tasks to frontier models.
• Dual-review for high-risk code/security decisions: two independent models produce findings, deterministic rules reconcile and human reviews disagreements.
• Shadow evaluation: send a sanitized copy of eligible tasks to a candidate model without using its output, then compare offline.

12.5 Router telemetry

model_decision:
  task_id: ...
  capability_class: frontier-code
  selected: codex
  reason_codes: ["repo_edit", "eval_best", "data_policy_allows_cloud"]
  alternatives_rejected:
    - local-code: "golden-task score below threshold for migration refactor"
  latency_ms: 84211
  cost_usd: 1.72
  outcome_score: 0.94

12.6 Final architecture rule

Do not build a “Claude agent,” a “Codex agent,” and a “local agent” as three unrelated systems. Build one enterprise agent architecture with provider-specific adapters. Tools, retrieval, identity, approvals, audit and evaluation stay stable while the model/harness can change. That is the difference between a useful pilot and an enterprise platform that can survive the model landscape of the next two years.

Implementation Blueprint: Policy First, Benchmark Second

Model registry

ModelRecord { id, provider, endpoint, deployment_version,
  capabilities:[code,tools,structured,vision], max_data_class, allowed_regions,
  max_context, latency_slo, unit_cost, eval_scores, runtime, health, capacity }

Routing stages

1. Hard policy filter: data class, geography, contractual restrictions and maximum agency.
2. Capability filter: code/tool use/structured output/context/vision.
3. Health/capacity filter: remove unhealthy or saturated deployments.
4. Task-quality threshold from your own eval store.
5. Optimize latency/cost among models that clear quality.
6. Generate a fallback chain at task start so failover cannot weaken data policy.
7. Record observed outcome/latency/cost for future routing analysis.

Operate local inference like a production service

• Pin weights, tokenizer and chat-template versions.
• Benchmark every quantization/runtime combination because tool calling can regress independently of benchmark quality.
• Measure queue depth, tokens/s, KV-cache utilization, GPU memory, batch size, TTFT and cancellation rate.
• Separate interactive and batch GPU pools.
• Use admission control and graceful downgrade rather than unbounded queues.

A correct route is the cheapest/fastest eligible model that clears quality and policy thresholds, not the model with the highest generic leaderboard score. Include provider outage, GPU saturation, restricted data, oversize context and missing tool capabilities in router evaluations.

Source and Shared Prerequisites

Adapted from the September 2026 Enterprise AI Agent Mesh handbook, Article 12 and Blueprint 12. The Enterprise Agent Platform Foundation guide provides the shared identity, MCP, retrieval, sandbox, audit and evaluation design, plus the source handbook’s further-reading list. Validate model, protocol and tool versions before production use.

Explore the Series

Previous guide · Series foundation