AI Model Router Agent — Engineering Guide

InfinitySDLC Engineering Guides · 12/12

A hybrid enterprise agent stack needs an explicit model-selection layer. Without it, provider choices leak into business prompts and policy, quality, latency, cost and fallback behavior become impossible to govern consistently.

The router decides which eligible inference deployment may serve a task. It must not grant tool permissions, reinterpret data classification or expand agency; those remain separate control-plane decisions.

Reference engineering design, not a report of a completed client deployment. Sections 12.1–12.6 and the implementation blueprint preserve the September 2026 Enterprise AI Agent Mesh handbook. Sections 12.7–12.12 add production recommendations. Model names, scores, weights, prices and infrastructure values are examples or point-in-time capabilities and must be revalidated before use.

Policy removes ineligible routes before optimization begins. A cheaper or stronger model is irrelevant when the task is not allowed to reach that endpoint.

The Header diagram is a conceptual router, not a provider benchmark. It illustrates policy and capability filtering across hosted coding harnesses and local open-weight deployments. Open the routing diagram at full size.

12.1 Routing dimensions

Keep these dimensions explicit in a normalized task envelope. A routing decision made from an opaque natural-language prompt cannot be reproduced or audited reliably.

12.2 Policy-first routing

# Handbook pattern — illustrative weights, not universal defaults
def route(task):
    candidates = registry.models_supporting(task.capabilities)
    candidates = [m for m in candidates if task.data_class <= m.max_data_class]
    candidates = [m for m in candidates if task.agency <= m.max_agency]
    scored = [(m, 0.45*eval_score(m, task.type) + 0.20*latency_score(m) + 0.20*cost_score(m) + 0.15*availability_score(m)) for m in candidates]
    return max(scored, key=lambda x: x[1])[0]

The important property is the ordering, not the sample coefficients: hard policy filtering happens before quality, latency or cost optimization. If restricted data is confined to an approved sovereign endpoint, an otherwise stronger hosted model is not a candidate.

Business agents should request capability classes such as frontier-code, local-reasoning or compact-extraction. The router resolves the current eligible deployment. The model route must remain separate from tool authority: choosing a coding harness does not grant Git write access or production credentials.

Current MCP authorization guidance requires resource-bound tokens and server-side audience validation. Routing therefore cannot be used as a shortcut around downstream authorization.

12.3 Model and harness tiers

These are routing tiers, not a permanent ranking. OpenAI currently describes gpt-oss-120b and gpt-oss-20b as open-weight reasoning models intended for local or data-center deployment; self-hosting cost still depends on infrastructure and operations.

For Codex integrations, OpenAI’s February 4, 2026 description of the Codex App Server documents a bidirectional JSON-RPC surface over the same harness used across Codex clients. Treat the harness revision as part of the evaluated deployment. Claude Code likewise combines model capability with an execution harness; Anthropic’s sandboxing guidance separates filesystem and network isolation.

12.4 Fallback and ensemble patterns

• Same-provider fallback: only when the fallback deployment passes the same policy and capability gates.
• Cross-provider fallback: only when data handling, geography, retention and contractual constraints still match.
• Local continuity fallback: preserves service during an external outage only for task classes where the local deployment cleared quality thresholds.
• Escalation: compact deployments handle low-risk work and escalate ambiguity or high-impact work to an eligible higher-capability route.
• Independent review: high-risk decisions may use two independently routed analyses followed by deterministic reconciliation and human review.
• Shadow evaluation: an eligible sanitized copy may be sent to a candidate deployment without consuming the candidate output in production.

Fallback is not permission expansion. Compute an eligible chain at task admission and recompute when policy, health or capacity changes.

12.5 Router telemetry

model_decision:
  task_id: task-017
  policy_revision: routing-policy-r19
  registry_revision: registry-2026-09-15
  capability_class: frontier-code
  selected_deployment: codex-approved-eu-r4
  selected_harness: codex-app-server-tested-build
  reason_codes: [repo_edit, quality_gate_passed, cloud_allowed]
  alternatives_rejected:
    - deployment: local-code-r7
      reason: quality_threshold_not_met
  outcome_ref: eval-or-production-result-id

Persist reason codes and revision identifiers, not sensitive prompts or raw retrieved context. A route should remain explainable after registry entries, prices and health states change. Separate routing telemetry from outcome telemetry: “eligible endpoint selected” and “task succeeded” are different assertions.

12.6 Final architecture rule

Do not build a “Claude agent,” a “Codex agent” and a “local agent” as unrelated systems. Build one enterprise agent architecture with provider- and harness-specific adapters. Identity, retrieval, tools, approvals, audit and evaluation remain stable while the model route changes.

Provider-specific capabilities still matter. Codex App Server exposes durable thread and approval primitives, while Claude Code provides its own sandbox and permission model. The control plane normalizes the enterprise contract without pretending the harnesses are identical.

12.7 Make the model registry versioned control-plane data

A model name is not an adequate routing record. The same family can exist behind different endpoints, regions, retention terms, quantizations, runtimes, tool adapters and harness versions. Register the deployable unit that was actually evaluated.

ModelDeployment {
  deployment_id, provider, model_family, model_revision,
  harness_id, harness_revision, endpoint_region,
  data_handling_profile, capabilities, max_context_contract,
  allowed_data_classes, max_agency_class,
  eval_bundle_revision, runtime_revision,
  health, capacity_class, lifecycle_state
}

Treat capability claims as testable contracts. “Structured output supported” means the exact deployment passed the schema behavior you depend on. “Tool calling supported” means the tested harness can select and execute your tool protocol correctly. Registry changes are configuration changes with blast radius: version, review and support rollback.

12.8 Separate eligibility from optimization

A production router is easier to reason about with two explicit phases. Eligibility is deterministic and fail-closed: data boundary, contractual restrictions, required capability, region, agency ceiling, deployment lifecycle and minimum eval revision. Optimization chooses among survivors using quality, deadline, queue, availability and cost.

Do not let model-generated text modify these fields. A retrieved document saying “send this to provider X” is content, not routing authority.

12.9 Promote deployments through task-specific evaluations

Evaluate the complete routed system: model revision, harness, system instructions, tool schemas, retrieval behavior and runtime. Swapping only the model can still change tool selection, refusal behavior and long-horizon consistency.

Anthropic’s January 2026 agent-evaluation guidance distinguishes the transcript from the actual environment outcome and recommends capability plus regression suites. Apply that distinction to routing: a candidate must solve the task and preserve required side effects, not merely produce convincing text.

1. Register the candidate as non-production and bind it to an immutable evaluation bundle.
2. Run capability tests for target task classes and regression tests for already-supported behavior.
3. Repeat nondeterministic tasks enough to measure consistency rather than one lucky success.
4. Inspect failures and grader quality; a broken eval must not block or promote a route.
5. Shadow eligible production traffic where policy permits.
6. Canary the route for a bounded task population.
7. Promote, hold or roll back the registry revision using predefined outcome gates.

Generic benchmarks can inform discovery; your versioned enterprise tasks determine production eligibility.

12.10 Treat capacity and admission control as routing inputs

A locally hosted deployment can be policy-eligible and still unavailable. Queue depth, memory pressure and accelerator capacity belong in admission control. Silent overload produces long latency and correlated failure.

For vLLM-based deployments, current production metrics expose queue time, time to first token, request phase times and KV-cache-related telemetry. Kubernetes exposes GPUs as schedulable resources through device plugins; its GPU scheduling documentation makes accelerator availability an infrastructure constraint rather than an LLM property.

Separate interactive and batch admission classes where their latency requirements conflict. Reject or queue excess work under bounded limits. An overloaded local endpoint is not a reason to send restricted data to an ineligible hosted service.

eligible = policy_filter(task, registry_revision)
quality_ok = eval_filter(eligible, task.eval_contract)
operable = capacity_filter(quality_ok, task.deadline)
if not quality_ok: return BLOCKED_NO_QUALITY_ROUTE
if not operable: return QUEUED_OR_EXPLICITLY_DEGRADED
return optimize(operable, objective=task.routing_objective)

12.11 Design fallback for state continuity and side-effect safety

Failing over a stateless classification call is straightforward. Failing over halfway through a coding or operational agent loop is not. Provider-specific conversation state, compaction, tool representations and approval primitives may not be portable.

Persist a provider-neutral checkpoint outside the model: normalized task, evidence references, tool-call ledger, produced artifacts, approval state and unresolved actions. On failover, resume from a safe checkpoint instead of assuming another harness can consume opaque provider state.

If a model call fails after a mutating tool request may have been submitted, the next route must reconcile authoritative tool state before retrying. Model failover never justifies replaying a deployment, ticket creation or merge blindly. Idempotency belongs to the tool contract and operation identifier.

OpenAI’s App Server architecture explicitly exposes thread, turn, item and approval lifecycles. Treat those as harness-specific semantics behind the adapter. Anthropic’s sandbox and permission model is likewise a harness boundary. The enterprise checkpoint references these states without pretending they are interchangeable.

12.12 Adversarial acceptance tests and operating metrics

Track ineligible-route attempts, policy-denied tasks, quality-gate failures, fallback frequency, route changes by registry revision, queue time, end-to-end latency, cost per verified successful task, local saturation, cancellation rate and regression rate after promotion. A zero routing-policy violation rate is a control objective; it is not proof that model outputs are correct.

Also test provider outages, stale health data, registry rollback, evaluator unavailability, oversize context and partial telemetry failure. When evidence is insufficient, return an explicit blocked or unknown state instead of fabricating a compliant route.

Implementation Blueprint: Policy First, Benchmark Second

Model registry

ModelRecord { deployment_id, provider, endpoint, deployment_version,
  harness_revision, capabilities, max_data_class, allowed_regions,
  max_context_contract, max_agency, latency_objectives, unit_cost_model,
  eval_bundle_revision, runtime_revision, health, capacity, lifecycle_state }

Routing stages

1. Hard policy filter: data class, geography, contractual restrictions and maximum agency.
2. Capability filter: code, tools, structured output, context and vision.
3. Health and capacity filter: remove deployments that cannot accept the task within policy.
4. Task-quality threshold from the versioned eval store.
5. Optimize deadline, expected quality and cost among surviving routes.
6. Generate the eligible fallback chain at task admission so failover cannot weaken policy.
7. Record the routing decision and verified outcome for later analysis.

Operate local inference like a production service

• Pin weights, tokenizer, chat template, runtime and adapter versions.
• Benchmark every quantization/runtime combination used in production because tool and structured-output behavior can regress independently of generic quality.
• Measure queue time, TTFT, token throughput, KV-cache pressure, accelerator memory, cancellations and admission denials.
• Separate interactive and batch pools or quotas where their SLOs conflict.
• Use bounded admission and explicit degradation instead of unbounded queues.

A correct route is the eligible deployment that clears the task’s quality contract and best satisfies the current routing objective. It is not necessarily the model with the highest generic benchmark, the lowest token price or the shortest current queue.

Sources and shared prerequisites

Adapted from Article 12 of the September 2026 Enterprise AI Agent Mesh handbook. The production sections above extend the handbook’s policy-first router with versioned deployment identity, evaluation promotion, capacity admission and side-effect-safe failover. The Enterprise Agent Platform Foundation supplies shared identity, MCP policy, retrieval, sandboxing, audit and evaluation infrastructure.

Primary implementation references used here include OpenAI’s Codex App Server architecture, Codex security guidance and open-weight model documentation; Anthropic’s Claude Code sandboxing and agent evaluation guidance; MCP authorization; vLLM production metrics; and Kubernetes GPU scheduling. Validate the exact deployment, contract and tool versions selected for production.

Explore the series

Previous: Incident Response Agent

Series foundation