LLM Model Routing, Cascades & Mixture-of-Agents

LLM Model Routing, Cascades & Mixture-of-Agents

• Reference under the ai-agent-engineering hub. The multi-model serving-decision layer - choosing/orchestrating WHICH model(s) answer each request to ride the cost/quality/latency Pareto frontier - distinct from serving ONE model well (that is llm-inference-serving). Full reference: references/llm-routing-cascades.md. [source]
• Three families, by when the decision happens: route = pick a model before generation (1 call); cascade = run cheap, observe, escalate on low confidence (1–N sequential); ensemble/mixture = run several and fuse (N parallel). Plus the zeroth route: cache = serve a remembered answer (0 calls). [source]
• Predictive routing (RouteLLM, arXiv:2406.18665) - pick a model before generation. Four router types: similarity-weighted ranking, matrix factorization (best on MT-Bench), BERT classifier, causal-LLM classifier. Trained on Chatbot-Arena preference data + LLM-judge/golden-label augmentation. Strong-vs-weak binary with a cost-quality threshold; metrics PGR (Performance Gap Recovered) and CPT (Call-Performance Threshold); ~85% cost cut on MT-Bench at 95% GPT-4. [source]
• Route-by-difficulty - Route-to-Reason (2505.19435, model+strategy under budget, −60% tokens), RADAR (multi-objective Pareto), adaptive think/non-think (reasoning models overthink easy queries). [source]
• Cascades + deferral/abstention (FrugalGPT, arXiv:2305.05176) - cheap-first, score, escalate; learned scorer + thresholds; up to 98% cost cut matching GPT-4. Calibration is the whole game (over/under-defer). Router (upfront, no feedback) vs cascade (observes cheap answer, pays latency). [source]
• Speculative cascades (arXiv:2405.19261, ICLR 2025) - token-level flexible deferral across two models. NOT speculative decoding: decoding is loss-less within one model (output identical); speculative cascade routes across models with a controlled quality change. [source]
• Mixture-of-Agents (MoA, arXiv:2406.04692) - layered proposers + aggregator; collaborativeness (better with others' outputs, even weaker ones); 65.1% AlpacaEval 2.0 LC (OSS) vs GPT-4o 57.5%. Cost: many calls + high TTFT; Self-MoA critique (one strong model resampled can win). [source]
• Output ensembling (LLM-Blender, arXiv:2306.02561) - PairRanker (rank candidates) + GenFuser (fuse top-K). Input-level (routing/MoA) vs output-level (run N, fuse once). [source]
• Semantic / prompt caching as routing (GPTCache) - embedding-similarity cache (paraphrases hit), ~68.8% call reduction; exact-match/prefix vs semantic; the cheapest "route" (0 calls). [source]
• Cost/quality/latency Pareto modeling - maintain a frontier across models; per-request constrained optimization (max quality s.t. cost/latency budget); routers push the frontier outward vs any single point. [source]
• Router evaluation (RouterBench, arXiv:2403.12031) - 405k pre-computed outcomes; evaluate on cost-quality curves / AIQ, not a single operating point; RouterArena (2510.00202). [source]
• Tooling - gateway (LiteLLM, OpenRouter transport: load-balance/fallback) vs quality-predictive (RouteLLM OSS, NotDiamond [powers OpenRouter Auto], Martian, vLLM Semantic Router 'Iris' v0.1 Jan-2026). [source]
• Failure modes - routing collapse (defaults to the expensive model as budget rises), tail miscalibration (rare high-stakes queries), added latency (cascade escalation, MoA TTFT), maintenance (re-fit on fleet/price change). [source]
• Boundaries: serving ONE model (vLLM/batching/KV/autoscaling) → llm-inference-serving; speculative DECODING within one model → llm-inference-serving; reasoning route-by-difficulty cost bullet → reasoning-models; agent orchestration / tool loops → agent-ecosystem/autonomous-loops (this is model SELECTION). [source]
• Primary sources: RouteLLM 2406.18665; FrugalGPT 2305.05176; Speculative cascades 2405.19261; MoA 2406.04692; LLM-Blender 2306.02561; RouterBench 2403.12031; Route-to-Reason 2505.19435; GPTCache; vLLM Semantic Router; "When Routing Collapses" 2602.03478. [source]