← narwal.one/Second Brain
SecondBrain
Ask the Brain
Index/Conceptupdated Sat Aug 08 2026 08:00:00 GMT+0800 (Philippine Standard Time)

Model Routing

model-routingcostefficiencymulti-modelenterprise-ai

Model Routing

Deliberately matching each task to the cheapest model that is sufficient for it, instead of running everything on the frontier model. The practical-economics counterweight to ever-more-capable (and ever-more-expensive) models like Fable 5.

The argument (Matthew Berman)

The launch of Fable 5 at $10/$50 per million tokens (≈2× Opus) is the cleanest case for routing yet: you don't need Fable for the vast majority of use cases.

"You have the absolute frontier with Fable and you give it your most difficult problems… and then for everything else, you don't need it. You can go back to Sonnet. You can go back to Haiku."

  • We're heading into a multi-model world; knowing which task goes to which model is a core skill.
  • Enterprises are "already getting crazy bills" from Anthropic and OpenAI — routing is the cost-control lever.
  • Effort levels are routing within a model: both creators advise starting at the lowest effort and dialing up only when needed (Fable on low ≈ Opus 4.8 on X-high, per Nate).
  • Practitioner datapoint (I Turned Claude Fable Into The Ultimate Second Brain (Nate Herk)): Fable ate a $200/mo Max plan's 5-hour session limit in ~1 hour of stress-testing; Nate's mitigation is exactly routing — keep Fable for the lead session, delegate parallel sub-tasks to "cheaper workers" (Sonnet/Haiku) and collect one clean summary back.

2026-07-06 — June 2026: routing gets a sovereignty rationale and named architectures

Per The Big Ways AI Just Changed (AI Daily Brief), the Fable-suspension weeks turned routing from a cost tactic into a two-dimensional diversification strategy — cost + sovereignty ("we now had both a cost and a sovereignty dimension for companies to think about diversifying their architecture away from just OpenAI or Anthropic"). Routing companies saw a surge of experimentation, and the month produced named production architectures beyond down-tiering:

  • Worker + advisor: Harvey + Fireworks paired an open-weight GLM 5.2 worker with an Opus advisor for legal tasks — better than Opus alone at a fraction of the cost.
  • Panel + judge + synthesizer: OpenRouter's Fusion runs a panel of models, a judge, and a synthesizer on hard tasks — SOTA-level capability claimed at lower cost.
  • Post-trained open-weight specialists: Cursor's Composer 2.5 (built off Kimi) as the buy-side version of owning your routed tier.

Local AI / open-weight fallbacks reached genuine boardroom-agenda status for the first time in the same window — routing is the architecture that makes that re-evaluation actionable.

2026-07-28 — "Model routing is a first-class feature now": the $10B consolidation beat

Per Why AI Hasnt Increased Unemployment According to Anthropic (AI Daily Brief), routing completes the arc from tactic → architecture → acquisition-grade product category:

  • Stripe in talks to acquire OpenRouter for ~$10B (WSJ) — ~7.7× OpenRouter's $1.3B valuation from two months earlier. The buy thesis (Macaroni Capital): "Stripe isn't buying an AI company. It's buying the metering and billing layer for inference plus the developer funnel attached to it." OpenRouter's inference margin is durable; routing is where Token Scarcity-era enterprise budgets get enforced.
  • Cursor Router ships — three optimization modes (intelligence / cost / balanced); claims Fable-level performance at 60% lower cost in intelligence mode (subjective-satisfaction metrics; early testers report no quality drop vs routing everything to Opus 4.8). Cursor CTO David Pan on the motivation: "We briefly went insane and decided every software engineer should also become an expert in model benchmarks, thinking levels, and cache hit rates." The routing-in-the-harness pattern removes model selection as a user decision entirely.
  • Everyone builds one: Meta (internal incubator), Ramp, Vercel routers all live. Matthew Berman closes his own 2026-06-10 framing on this page: "Model routing is a first-class feature now."
  • Supply-side mirror: Microsoft's Nadella describes MAI strategy in explicitly routing terms — "routing to the MAI models for lower-end tasks… while continuing to use frontier models for frontier needs." Routing is now both a buyer discipline and a vendor architecture.

2026-08-08 — Cache-hit-rate is the real cost lever, not per-token headline price

Per the DeepSeek-V4-Flash-0731 Reddit Reception Research (2026-08-08) corpus (r/LocalLLaMA + r/LLMDevs, 684 comments), the per-token pricing gap between DeepSeek V4 Flash ($0.14 in / $0.28 out) and Anthropic Luna ($0.20 in / $1.20 out) is meaningful but is not what actually decides routed-tier economics. The cache-hit discount does — DeepSeek 98% vs Luna 90%, and real-world hit rates users measured are 90–99.6% (u/thecstep: 99.6% on 10B+ tokens). Once cached input is billed at 2% of list, the routed workload's effective cost is dominated by output tokens on cache-hit calls, which is where DeepSeek's 4×-cheaper output rate compounds with the 5-point cache advantage.

Two operational corollaries for routing:

  • A router that doesn't measure cache-hit-rate per (model × harness × workload) is under-instrumented. The vault has been carrying Token Scarcity's "feedback-per-token" reframing on the Effective Feedback Compute axis; cache-hit-rate is the pricing-side companion metric. Per-token list pricing is now the wrong denominator for anything but rough sizing.
  • The routed lower tier is now empirically an API call, not a rack. Per Cost-Structure Inversion of Local Inference, even r/LocalLLaMA members with the hardware openly admit the routed-tier open-weight capability is cheaper to reach via OpenRouter (>150 tok/s at ~$0.14/$0.28) than to self-host. The "own it to route to it" architectural option has been effectively priced out for most workloads. Data-residency, offline, or fine-tuning use cases still favor self-hosted; general routing does not.

Connects to the rest of the vault

  • Token Maxing — routing is the demand-side mitigation for runaway token spend; pairs with the EFC "feedback-per-token" reframing. Don't pay frontier rates for work a mid-tier model nails.
  • Build vs Buy (Agents) / enterprise cost discipline — Praveen's "decompose costs, tie to ROI" advice operationalizes the same instinct at org scale.
  • Mythos-Class Models — the frontier tier that makes routing economically urgent (50× the output price of Haiku-class work).

Sources

  • MYTHOS MYTHOS MYTHOS (Matthew Berman) (canonical for this framing)
  • Claude Mythos is Finally Here (Nate Herk) (effort-level routing)