Route every prompt to the right model — and cut AI cloud costs by up to 90%

A frontier model is priced for its worst query. The per-token rate has to cover the hard cases — multi-step reasoning, long-context synthesis, gnarly code. But your production traffic does not look like that. It is mostly easy: a one-line clarification, a yes/no classification, a JSON reformat, an autocomplete, a boilerplate function. On that traffic a small open model runs indistinguishably from a frontier API to the user — and it costs nothing per token beyond hardware you already own.

Routing every request to the frontier model spends the frontier price on a stream whose median query never needed it. That is the single largest avoidable line item in an LLM product, and it is avoidable because the mismatch is structural: the price is set by the tail, the volume is in the head. Hanzo Router turns that mismatch into a saving by sending each request to the cheapest model that can still serve it.

How it works — local-first, cheapest that serves

The idea is deliberately simple. For each request the router walks a short, ordered preference:

1. Reuse a local model already resident and serving — zero marginal cost, wins outright.
2. Load local — the best open model that still fits your available memory.
3. Cloud — fall through to the cheapest cloud model that can do the job.
4. Never silently degrade — if nothing suitable is available, the caller gets a clear error instead of a bad answer.

"Run it locally if the memory is there, else the next best wherever it is." Placement is separated from resolution: the router decides which class of model a request needs (local open, cheap cloud, or frontier), and the gateway decides how to reach and price the specific model it lands on — so cost tiering and provider redundancy stay cleanly apart.

The policy is declarative. Per task, an ordered prefer list of model ids (first usable wins) leads with your open Zen models so short prompts are answered locally for free, with a frontier model kept last for the hard cases a small model would miss, and a global cost ceiling that keeps anything too pricey out entirely:

# router policy — cheap-chat and code lead with local Zen models.
prefer:
  cheap_chat: [zen-nano, gpt-cheap-tier]
  code:       [zen-coder, frontier-code]
  general:    [zen-agent, cheap-cloud]
cost_ceiling: 3.00   # never route to a cloud model dearer than this (per 1k)

Enso — the learned layer (Hanzo Cloud)

Placement is open mechanism. Which model is genuinely best for this request and this user is policy — and policy is where learning belongs. That learned layer is Enso, Hanzo's proprietary model-orchestration model, available on Hanzo Cloud.

At a high level: a simple rule-based policy handles cold start (and is always the floor everything can fall back to), and Enso improves on it by ranking candidate models by expected quality against cost and latency under your service-level objective — respecting hard ceilings you set, and adapting to each user's taste over time. You keep control of the pool: opt specific providers or models out to satisfy data, privacy, or compliance requirements, with a full trace of which model actually ran. The mechanism above is the same either way — Enso just makes a smarter choice within it. (The learned internals aren't open; the placement layer and the Zen models are.)

Qualitative results — what it feels like in production

• Simple traffic is instant and free. Chat, classification, and formatting are answered by a resident local model with no provider round-trip and no per-token bill.
• Hard traffic still gets the frontier. A quality floor is enforced, not bargained: a request that needs deep reasoning is never quietly served by a model below the bar, so aggressive downgrading can't silently regress quality.
• It degrades gracefully. When a provider nears its rate limit or a device is out of memory, the request slides to the next best option instead of failing.
• One integration, one bill. Everything is one OpenAI-compatible endpoint and one metering plane — no juggling a stack of per-provider SDKs and invoices.

Quantitative results — the cost arithmetic, worked not fabricated

We do not report a measured "90%" benchmark, because we have not run a controlled A/B at a scale that would make that number honest. Instead, here is the arithmetic — substitute your own mix and prices.

Take one million requests, each averaging 1,000 input and 300 output tokens.

• Frontier at $3.00 / $15.00 per million tokens → 1000/1e6·3 + 300/1e6·15 = $0.0075 per request.
• Cheap cloud tier at $0.15 / $0.60 per million → $0.00033 per request — about 1/23 of frontier.
• Local open model → no per-token API charge. Not literally free — it draws electricity, accounted below — but you pay no provider per token.

Segment the stream by what each request actually needs:

Segment	Share	Routed to	$/req	API cost / 1M reqs
Simple (chat, classify, format)	70%	local (open)	0.00000	$0
Medium (general, routine code)	20%	cheap cloud	0.00033	$66
Hard (reasoning, long-context)	10%	frontier	0.00750	$750
Routed total				$816
Baseline: all-frontier	100%	frontier	0.00750	$7,500

Routed API spend is $816 against $7,500 all-frontier — a 1 − 816/7500 = 89.1% reduction. The "up to 90%" is exactly this arithmetic, and it moves the way you'd expect: at an 80/15/5 split it's 94%; a stream with no simple traffic saves nothing, because there is nothing cheap to route to. The lever is the share of requests that don't need the frontier model — and in real assistant and agent traffic, that share is the majority. This is the same economics the routing and cascade literature (RouteLLM, FrugalGPT) reports.

Total cost of ownership: a 20-person team

The bill is more than tokens. Local-first routing and one usage plane move four more lines — compute (local is cheap, not free), avoided per-seat subscriptions, avoided provider overage tiers, and simpler ops. Same one-million-request workload, monthly, against a frontier-default API plus per-seat assistant subs:

Monthly cost line (20-person team)	Status quo	With router
API token spend (table above)	$7,500	$816
Local-tier compute (electricity)	$0	$70
Per-seat assistant subs (20 × $20)	$400	$0
Provider overage / burst tiers	$150	$0
Routing + metering fee (illustrative 1%)	$0	$8
Monthly total (metered bill)	$8,050	$894

That's $8,050 → $894, an ≈ 88.9% reduction — and local-first additionally avoids the ~$580/month a rented GPU would cost to serve the same local tier. The headline stays ~89% because tokens dominate, but the itemization is the point: compute, seats, overage, and ops all move the same direction. Every figure here is a labeled assumption — swap in your own.

The honest version

The saving is not magic and we won't dress it as a benchmark: it is the arithmetic of not paying the frontier price on a stream whose median request never needed it. Aggressive downgrading risks a quality regression — the enforced quality floor is the guard. Turn it on, watch the floor, and let the mix do the rest.

• Enso (learned orchestration, on Hanzo Cloud): hanzo.ai/enso
• Zen (open-weight models, run anywhere): hanzo.ai/enso
• Usage plane: docs.hanzo.ai/docs/usage