AI Gateways & LLM Proxy Infrastructure
Parent: LLM Models and APIs · researched 2026-06-03T22:51:09.481Z· 14 sources · 12 concepts · skill llm-ai-gateways
An AI gateway (LLM gateway / LLM proxy) is the production traffic-and-control
AI Gateways & LLM Proxy Infrastructure
- An AI gateway (LLM gateway / LLM proxy) is the production traffic-and-control [source]
- plane between your applications and one-or-many LLM providers. It exposes a single, [source]
- usually OpenAI-compatible API while centrally enforcing key management, rate [source]
- limits, budgets, fallbacks, caching, guardrails, observability, and governance. [source]
- The category exists because traditional API gateways cannot count tokens, cannot [source]
- treat streaming (SSE) responses as first-class, cannot enforce content-level [source]
- security, and have no notion of multi-provider model credentials or token-cost [source]
- attribution. This skill is the ops/governance proxy plane - not the algorithm [source]
- that decides which model answers a request. [source]
When to use / Skip
- Use this skill when you are: [source]
- choosing whether to front LLM providers with a gateway, and which product; [source]
- issuing virtual keys to teams/apps instead of distributing raw provider keys; [source]
- enforcing rate limits, budgets, spend tracking, and chargeback; [source]
- building fallback/retry/load-balancing for provider reliability (not cost-optimal model picking); [source]
- configuring caching, PII/guardrails, observability, RBAC/audit at the proxy; [source]
- deciding self-hosted vs managed vs platform-native deployment. [source]
- Skip to a peer when the task is: [source]
- WHICH model answers - predictive routing (RouteLLM), model cascades & deferral [source]
- (FrugalGPT), speculative cascades, mixture-of-agents, route-by-difficulty, and the [source]
- semantic-cache similarity/threshold internals (GPTCache) -> llm-routing-cascades. [source]
- (LiteLLM/OpenRouter appear in both; here they are gateway products.) [source]
- Serving one model for throughput/latency (vLLM, PagedAttention, batching) -> llm-inference-serving. [source]
- Generic, non-AI API gateway design (versioning, REST/GraphQL) -> software-engineering-patterns. [source]
The AI-gateway control plane
Caching at the gateway (and the boundary)
- Gateways implement exact-match caching with an explicit cache key + TTL: [source]
- Cloudflare - cf-aig-cache-key + cf-aig-cache-ttl (min 60s, max 1 month); [source]
- default 5-min if caching enabled; cache is volatile (concurrent identical [source]
- requests can race and miss). [source]
- Helicone - Cloudflare Workers KV, Cache-Control: max-age (default 7d, [source]
- max 365d), bucket size <= 20. [source]
- Some gateways add semantic caching (embedding-similarity hits on [source]
- differently-worded but same-meaning prompts): Portkey (simple + semantic), [source]
- Kong AI Semantic Cache (3.8; embeddings generated on the fly, stored in [source]
- Redis or Postgres). At the gateway you decide *where the cache sits, the key, [source]
- TTL, and how it interacts with cost attribution*. For the **similarity/threshold [source]
- internals and the algorithmic risk of false hits (GPTCache et al.)** -> [source]
- llm-routing-cascades. [source]
Vendor landscape
LiteLLM Proxy (LLM Gateway) — OSS, self-hosted-first
- Identity: canonical OSS gateway (Python, MIT) + enterprise tier. [source]
- API: 100+ providers in OpenAI ChatCompletions/Completions; Anthropic SDK [source]
- + MCP/agent gateway. Endpoints: /chat/completions, /embeddings, /models, [source]
- Control plane: /key/generate virtual keys with max_budget, [source]
- budget_duration, tpm_limit, rpm_limit, max_parallel_requests; spend [source]
- auto-tracked in LiteLLM_VerificationToken/UserTable/TeamTable; [source]
- fallbacks (fallbacks, context_window_fallbacks, content_policy_fallbacks) [source]
- + num_retries + cooldown_time; LB (simple-shuffle, least-busy, [source]
- usage/latency/cost-based), Redis for multi-instance limits; key rotation [source]
- with grace period. Caching + guardrails (incl. PII). [source]
- Enterprise gates: secret managers (**Azure Key Vault, Google Secret Manager, [source]
- HashiCorp Vault, CyberArk Conjur, AWS Secrets Manager**), SSO/SAML, audit logs, [source]
- multi-team. Prometheus/OTel/Langfuse. [source]
- Scale/deploy: load-tested 1.5k+ req/s; self-host (Docker) or LiteLLM Cloud. [source]
Portkey — OSS gateway + managed/enterprise
- Identity: npx @portkey-ai/gateway, 1600-3000+ models, 50+ contributors. [source]
- API: Universal API; JSON Configs set strategy.mode = [source]
- single/fallback/loadbalance/conditional (Zod-validated). [source]
- Control plane: virtual keys in a vault; budget limits; hourly/daily/ [source]
- per-minute rate limits; simple + semantic cache; circuit breaker; [source]
- 40+ guardrails incl. PII/PHI redaction + org-level enforcement; RBAC; [source]
- org-wide audit logs; SOC2/HIPAA/GDPR/CCPA. [source]
- Deploy: SaaS, hybrid (gateway + data-plane in your VPC, control plane at [source]
- Portkey), or fully air-gapped. Observability retention 3d/30d/custom. [source]
Cloudflare AI Gateway — managed, edge, free core
- Identity: edge-deployed; core features free (one line of code). [source]
- API: OpenAI-compatible unified endpoint; Unified Billing or BYOK. [source]
- Control plane: caching (above); rate limiting fixed/sliding -> 429; [source]
- Dynamic Routing - visual/JSON flows with Conditional / Percentage / Model / [source]
- Rate Limit / Budget Limit nodes, versioned + rollback; Guardrails via [source]
- @cf/meta/llama-guard-3-8b (billed as Workers AI tokens); DLP in the Firewall. [source]
- Limits (2025): logs free 100K/mo, paid 1M/mo, ~10M/gateway cap; Workers [source]
Kong AI Gateway — plugins on Kong Gateway
- Identity: plugin set on Kong Gateway (self-hosted or Konnect SaaS). [source]
- API: ai-proxy/ai-proxy-advanced normalize providers to llm/v1/chat. [source]
- Control plane: AI Semantic Cache (3.8, Redis/Postgres); **6 LLM LB [source]
- algorithms (3.8); AI (Semantic) Prompt Guard allow/deny -> 4xx; 3.10** [source]
- RAG injection + PII sanitization; full enterprise API-gateway authn/rate-limit/RBAC. [source]
Helicone — OSS (Rust), observability-first that also proxies
- Identity: Rust, GPL-3.0; single endpoint, 100+ providers; 0% markup [source]
- Control plane: smart routing (model-latency, P2C+PeakEWMA, weighted, cost); [source]
- rate limits per user/team/global (requests/tokens/dollars); caching on [source]
- Workers KV (default 7d, max 365d) or self-host Redis/S3; sessions, prompt [source]
- Self-host: Web(3000) + Jawn API/proxy(8585) + Worker + **Postgres + [source]
- ClickHouse + MinIO/S3 + Redis. WARNING: port 8585 has no auth by default.** [source]
TrueFoundry — enterprise gateway, low-latency
- Identity: sub-3ms internal latency at enterprise scale. [source]
- Control plane: rate limiting + token budgeting, per-user/app/tool quotas; [source]
- guardrails (PII, toxicity); RBAC + per-team keys; MCP gateway (OAuth2/ [source]
- RBAC/metadata per tool call). [source]
- Deploy: SaaS, hybrid, or self-hosted/on-prem/air-gapped/multi-cloud; [source]
- stateless gateway pods + NATS + Postgres + ClickHouse. [source]
OpenRouter — marketplace/aggregator used as a gateway
- Identity: 315+ models, OpenAI-compatible (base https://openrouter.ai/api/v1). [source]
- Control plane: automatic fallback across providers/models (pay only for [source]
- successful runs); thinner governance/observability than dedicated gateways. [source]
- Pricing (mid-2026, per a review summary - verify on the pricing page): catalog [source]
- matches provider; PAYG ~5.5% fee; BYOK first 1M req/mo free, then 5%; free [source]
- models with ~20 RPM/200-per-day limits. [source]
Vercel AI Gateway — managed, GA
- Identity: unified HTTP API to hundreds of models, one key; GA (2025). [source]
- Control plane: no markup (pay provider price); BYOK zero fee; **$5/mo [source]
- free credits; automatic retry to other providers; Custom Reporting API** (beta) [source]
- — cost/token/request by model/provider/user-ID/tag/credential, incl. BYOK. [source]
- Plans (2025/26): Hobby $0 / Pro $20 / Enterprise custom; tight AI-SDK integration. [source]
AWS Bedrock gateway — a *pattern*, not a product
- Bedrock is not OpenAI-compatible itself. Patterns: [source]
- Bedrock Access Gateway (aws-samples/bedrock-access-gateway) - OSS [source]
- OpenAI-compat shim; deploy API Gateway + Lambda (Lambda Web Adapter for SSE, [source]
- no VPC, <=10-min timeout) or ALB + Fargate (lowest streaming latency, no cold [source]
- starts); Application Inference Profiles for cost tracking; prompt caching [source]
- (Claude/Nova, up to 90% cost / 85% latency). [source]
- LiteLLM/Portkey in front of Bedrock for full multi-provider governance. [source]
Databricks AI Gateway — evolving rebrand (CONTESTED naming/GA)
- Lineage: **MLflow AI Gateway / MLflow Deployments Server -> Mosaic AI Gateway [source]
- -> Unity AI Gateway. Primary doc (updated May 2026) says Unity AI Gateway**; [source]
- Data+AI Summit 2025 blog says "Mosaic AI Gateway"; another blog claims "Agent Bricks [source]
- AI Gateway is GA" - sources disagree; treat GA as unsettled. New LLM/agent/MCP [source]
- surface is Beta as of May 2026 (no charges during Beta). [source]
- Governs: LLM endpoints, agents, MCP servers, coding agents (Cursor / Claude [source]
- Code / Codex CLI / Gemini CLI). [source]
- Control plane: usage tracking, payload logging to Unity Catalog **inference [source]
- tables, per-user/group rate limits, guardrails (PII detection, safety/content [source]
- filtering), traffic splitting, cost via billable-usage system tables**. [source]
- Deploy: part of the Databricks lakehouse platform. [source]
Selection / decision guidance
- Self-hosted, full control, OSS, data never leaves your VPC -> LiteLLM [source]
- (broadest provider + key-vault + budget surface) or Portkey OSS (config-driven [source]
- routing + guardrails). Helicone if observability is the primary need. [source]
- Zero-ops managed, edge latency, free to start -> Cloudflare AI Gateway [source]
- (caching + dynamic routing + DLP) or Vercel AI Gateway (no-markup + AI-SDK apps). [source]
- Already on an enterprise API gateway -> Kong AI Gateway (reuse Kong RBAC/ [source]
- authn/rate-limit; add AI plugins). [source]
- Enterprise governance, hybrid/air-gapped, lowest latency, MCP governance -> [source]
- TrueFoundry or Portkey enterprise. [source]
- All-in on a cloud/data platform -> AWS Bedrock Access Gateway (AWS-native, [source]
- serverless) or Databricks Unity AI Gateway (lakehouse-native, Unity Catalog [source]
- audit) - accept the platform lock-in for native cost/audit tables. [source]
- Cheapest path to many models, light governance -> OpenRouter (accept the [source]
- ~5.5% fee and thinner controls). [source]
- Decision axes: OSS vs managed; data residency (does prompt data leave your [source]
- network?); markup model (flat fee vs no-markup vs infra-only); native key-vault vs [source]
- BYOK; guardrail depth (deterministic vs LLM-judge vs PII redaction); audit/RBAC [source]
- maturity; streaming fidelity; added latency hop tolerance. [source]
Integration patterns
Anti-patterns & failure modes
- Unauthenticated self-hosted proxy. Helicone's self-host **port 8585 has no auth [source]
- by default** - anyone with network access can proxy through it and burn your spend. [source]
- Firewall or add auth before exposing any self-hosted gateway. [source]
- Semantic-cache false hits. Two prompts with *similar embeddings but different [source]
- intent* return the same cached answer - a correctness bug, not just a stale-cache [source]
- bug. Threshold tuning is the real defense -> llm-routing-cascades. [source]
- Volatile / racing cache. Cloudflare cache is volatile: simultaneous identical [source]
- requests can both miss. Don't assume a write-then-read is atomic. [source]
- Fallback chains masking degradation. Silent fallback to a weaker model keeps the [source]
- service "up" while answer quality quietly drops - alert on fallback rate, not just [source]
- error rate. And retries can amplify load against an already-rate-limited provider [source]
- (retry storms) - use cooldowns + jitter. [source]
- Gateway as a single point of failure + latency hop. Every request now traverses [source]
- one component; an outage there takes down all providers at once. The flip side of [source]
- TrueFoundry's sub-3ms pitch - measure the added hop and run the gateway HA. [source]
- Cost-attribution blind spots under BYOK. When the provider bills you directly [source]
- (BYOK), the gateway may not see true spend - reconcile gateway spend tables against [source]
- provider invoices, and prefer gateways that report BYOK traffic (e.g., Vercel [source]
- Leaking raw provider keys. Distributing the real OPENAI_API_KEY to every app [source]
- defeats the gateway - issue virtual keys and keep provider keys in the vault. [source]
- Treating the gateway as a model router. Reliability fallback != cost/quality [source]
- model selection; don't hand-roll routing logic in the proxy when the discipline [source]
- lives in llm-routing-cascades. [source]
2025-2026 frontier
- OpenAI-compat surface is widening (unevenly). Chat Completions is the [source]
- universal contract; Responses API support is emerging and inconsistent [source]
- across gateways; several add Anthropic /v1/messages and MCP passthrough [source]
- (LiteLLM, TrueFoundry, Databricks). Verify per-vendor before assuming Responses works. [source]
- The gateway is becoming the agent/MCP control plane. Databricks Unity AI Gateway, [source]
- TrueFoundry, and LiteLLM now govern MCP servers, tool calls, and coding agents [source]
- (Cursor/Claude Code/Codex) - RBAC and audit applied per tool call, not just per [source]
- Guardrails moving inline + LLM-judge based. Cloudflare runs llama-guard-3-8b [source]
- inline; Portkey ships 40+ guardrails; Kong added on-the-fly embeddings for semantic [source]
- prompt-guard and PII sanitization. PII/PHI redaction at the proxy is now table-stakes [source]
- for regulated workloads. [source]
- Pricing models are bifurcating: no-markup / infra-only (Vercel, Helicone OSS, [source]
- Cloudflare core-free) vs flat platform fee (OpenRouter ~5.5%). BYOK with zero or [source]
- low fee is the competitive wedge. [source]
- Edge + Rust for latency. Helicone (Rust) and Cloudflare (edge Workers) chase the [source]
- "gateway adds no latency" promise; sub-3ms internal overhead (TrueFoundry) is now a [source]
- marketed differentiator. [source]
- Platform-native governance via system tables. Databricks (billable-usage + [source]
- inference tables in Unity Catalog) and AWS (Application Inference Profiles) fold cost [source]
- attribution and audit into the data/cloud platform itself. [source]
Sources
- LiteLLM - AI Gateway (LLM Proxy) overview: https://docs.litellm.ai/docs/simple_proxy [source]
- LiteLLM - Virtual Keys / Budgets / Rate Limits / Fallbacks / Load Balancing / Secret Managers: https://docs.litellm.ai/docs/proxy/virtual_keys , /docs/proxy/users , /docs/proxy/reliability , /docs/proxy/load_balancing , /docs/secret [source]
- Portkey - AI Gateway product + Configs + feature comparison + OSS gateway (npm): https://docs.portkey.ai/docs/product/ai-gateway , https://www.npmjs.com/package/@portkey-ai/gateway [source]
- Cloudflare AI Gateway - Overview / Features / Caching / Rate limiting / Dynamic routing: https://developers.cloudflare.com/ai-gateway/ [source]
- Cloudflare AI Gateway - Pricing & Limits: https://developers.cloudflare.com/ai-gateway/reference/pricing/ , /reference/limits/ [source]
- Kong AI Gateway - docs + 3.8 + 3.10: https://developer.konghq.com/ai-gateway/ , https://konghq.com/blog/product-releases/ai-gateway-3-8 , https://konghq.com/blog/product-releases/ai-gateway-3-10 [source]
- Kong - API Gateway vs AI Gateway: https://konghq.com/blog/learning-center/api-gateway-vs--ai-gateway [source]
- Helicone - AI Gateway overview + caching + self-hosting + GitHub: https://docs.helicone.ai/gateway/overview , https://docs.helicone.ai/features/advanced-usage/caching , https://github.com/helicone/ai-gateway [source]
- TrueFoundry - AI Gateway product + on-prem guide: https://www.truefoundry.com/ai-gateway , https://www.truefoundry.com/blog/ai-gateway-on-premise [source]
- OpenRouter - pricing + provider routing + FAQ: https://openrouter.ai/pricing , https://openrouter.ai/docs/guides/routing/provider-selection , https://openrouter.ai/docs/faq [source]
- Vercel AI Gateway - docs + pricing + BYOK + GA changelog: https://vercel.com/docs/ai-gateway , /docs/ai-gateway/pricing , https://vercel.com/changelog/ai-gateway-is-now-generally-available [source]
- AWS Bedrock Access Gateway: https://github.com/aws-samples/bedrock-access-gateway [source]
- Databricks Unity AI Gateway (Beta May 2026; lineage Mosaic/MLflow): https://docs.databricks.com/aws/en/ai-gateway/ , https://www.databricks.com/product/artificial-intelligence/ai-gateway [source]
- Category definition / why AI gateways emerged: https://atlan.com/know/what-is-ai-gateway-llm-gateway/ [source]
- > Boundary note: routing algorithms and semantic-cache internals defer to llm-routing-cascades; single-model serving to llm-inference-serving. Pricing and the Databricks gateway naming/GA are flagged as fast-moving / contested - re-verify before quoting. [source]
Children
- Unified OpenAI-compatible API surface (frontier)
- Virtual keys & key-vault management (BYOK) (frontier)
- Rate limiting & quotas (frontier)
- Budget/spend controls & cost attribution/chargeback (frontier)
- Fallback chains, retries & load balancing (frontier)
- Gateway caching (exact-match vs semantic) (frontier)
- Guardrail & PII/DLP enforcement at the proxy (frontier)
- Observability, logging & tracing hooks (frontier)
- Governance, RBAC & audit (frontier)
- Self-hosted vs managed vs edge vs platform-native (frontier)
- Streaming (SSE) pass-through (frontier)
- MCP/agent/tool-call gateway governance (frontier)
Frontier under this node: Budget/spend controls & cost attribution/chargeback, Fallback chains, retries & load balancing, Gateway caching (exact-match vs semantic), Governance, RBAC & audit, Guardrail & PII/DLP enforcement at the proxy, MCP/agent/tool-call gateway governance, Observability, logging & tracing hooks, Rate limiting & quotas, Self-hosted vs managed vs edge vs platform-native, Streaming (SSE) pass-through, Unified OpenAI-compatible API surface, Virtual keys & key-vault management (BYOK)