AI Gateways & LLM Proxy Infrastructure
Parent: LLM Models and APIs · Published reference · snapshot 2026-09-08
↓ Facts as markdownall context files
An AI gateway (LLM gateway / LLM proxy) is the production traffic-and-control
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
AI Gateways & LLM Proxy Infrastructure
- An AI gateway (LLM gateway / LLM proxy) is the production traffic-and-control plane between your applications and one-or-many LLM providers. It exposes a single, usually OpenAI-compatible API while centrally enforcing key management, rate limits, budgets, fallbacks, caching, guardrails, observability, and governance. The category exists because traditional API gateways cannot count tokens, cannot treat streaming (SSE) responses as first-class, cannot enforce content-level security, and have no notion of multi-provider model credentials or token-cost attribution. This skill is the ops/governance proxy plane - not the algorithm that decides which model answers a request. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#ai-gateways-llm-proxy-infrastructure]
When to use / Skip
- Use this skill when you are: [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#when-to-use-skip]
- choosing whether to front LLM providers with a gateway, and which product; [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#when-to-use-skip]
- issuing virtual keys to teams/apps instead of distributing raw provider keys; [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#when-to-use-skip]
- enforcing rate limits, budgets, spend tracking, and chargeback; [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#when-to-use-skip]
- building fallback/retry/load-balancing for provider reliability (not cost-optimal model picking); [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#when-to-use-skip]
- configuring caching, PII/guardrails, observability, RBAC/audit at the proxy; [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#when-to-use-skip]
- deciding self-hosted vs managed vs platform-native deployment. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#when-to-use-skip]
- Skip to a peer when the task is: [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#when-to-use-skip]
- WHICH model answers - predictive routing (RouteLLM), model cascades & deferral (FrugalGPT), speculative cascades, mixture-of-agents, route-by-difficulty, and the semantic-cache similarity/threshold internals (GPTCache) -> llm-routing-cascades. (LiteLLM/OpenRouter appear in both; here they are gateway products.) [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#when-to-use-skip]
- Serving one model for throughput/latency (vLLM, PagedAttention, batching) -> llm-inference-serving. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#when-to-use-skip]
- Generic, non-AI API gateway design (versioning, REST/GraphQL) -> software-engineering-patterns. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#when-to-use-skip]
The AI-gateway control plane
- A gateway is measured against this feature taxonomy. Each row is a column to fill when you compare vendors. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#the-ai-gateway-control-plane]
Caching at the gateway (and the boundary)
- Gateways implement exact-match caching with an explicit cache key + TTL: [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#caching-at-the-gateway-and-the-boundary]
- Cloudflare - cf-aig-cache-key + cf-aig-cache-ttl (min 60s, max 1 month); default 5-min if caching enabled; cache is volatile (concurrent identical requests can race and miss). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#caching-at-the-gateway-and-the-boundary]
- Helicone - Cloudflare Workers KV, Cache-Control: max-age (default 7d, max 365d), bucket size <= 20. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#caching-at-the-gateway-and-the-boundary]
- Some gateways add semantic caching (embedding-similarity hits on differently-worded but same-meaning prompts): Portkey (simple + semantic), Kong AI Semantic Cache (3.8; embeddings generated on the fly, stored in Redis or Postgres). At the gateway you decide where the cache sits, the key, TTL, and how it interacts with cost attribution. For the similarity/threshold internals and the algorithmic risk of false hits (GPTCache et al.) -> llm-routing-cascades. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#caching-at-the-gateway-and-the-boundary]
Vendor landscape
- > Prices/limits churn fast - every figure below is as documented in 2025-2026; > re-verify against the primary source before quoting to a customer. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#vendor-landscape]
LiteLLM Proxy (LLM Gateway) — OSS, self-hosted-first
- Identity: canonical OSS gateway (Python, MIT) + enterprise tier. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#litellm-proxy-llm-gateway-oss-self-hosted-first]
- API: 100+ providers in OpenAI ChatCompletions/Completions; Anthropic SDK + MCP/agent gateway. Endpoints: /chat/completions, /embeddings, /models, /key/generate. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#litellm-proxy-llm-gateway-oss-self-hosted-first]
- Control plane: /key/generate virtual keys with max_budget, budget_duration, tpm_limit, rpm_limit, max_parallel_requests; spend auto-tracked in LiteLLM_VerificationToken/UserTable/TeamTable; fallbacks (fallbacks, context_window_fallbacks, content_policy_fallbacks) + num_retries + cooldown_time; LB (simple-shuffle, least-busy, usage/latency/cost-based), Redis for multi-instance limits; key rotation with grace period. Caching + guardrails (incl. PII). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#litellm-proxy-llm-gateway-oss-self-hosted-first]
- Enterprise gates: secret managers (Azure Key Vault, Google Secret Manager, HashiCorp Vault, CyberArk Conjur, AWS Secrets Manager), SSO/SAML, audit logs, multi-team. Prometheus/OTel/Langfuse. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#litellm-proxy-llm-gateway-oss-self-hosted-first]
- Scale/deploy: load-tested 1.5k+ req/s; self-host (Docker) or LiteLLM Cloud. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#litellm-proxy-llm-gateway-oss-self-hosted-first]
Portkey — OSS gateway + managed/enterprise
- Identity: npx @portkey-ai/gateway, 1600-3000+ models, 50+ contributors. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#portkey-oss-gateway-managedenterprise]
- API: Universal API; JSON Configs set strategy.mode = single/fallback/loadbalance/conditional (Zod-validated). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#portkey-oss-gateway-managedenterprise]
- Control plane: virtual keys in a vault; budget limits; hourly/daily/ per-minute rate limits; simple + semantic cache; circuit breaker; 40+ guardrails incl. PII/PHI redaction + org-level enforcement; RBAC; org-wide audit logs; SOC2/HIPAA/GDPR/CCPA. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#portkey-oss-gateway-managedenterprise]
- Deploy: SaaS, hybrid (gateway + data-plane in your VPC, control plane at Portkey), or fully air-gapped. Observability retention 3d/30d/custom. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#portkey-oss-gateway-managedenterprise]
Cloudflare AI Gateway — managed, edge, free core
- Identity: edge-deployed; core features free (one line of code). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#cloudflare-ai-gateway-managed-edge-free-core]
- API: OpenAI-compatible unified endpoint; Unified Billing or BYOK. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#cloudflare-ai-gateway-managed-edge-free-core]
- Control plane: caching (above); rate limiting fixed/sliding -> 429; Dynamic Routing - visual/JSON flows with Conditional / Percentage / Model / Rate Limit / Budget Limit nodes, versioned + rollback; Guardrails via @cf/meta/llama-guard-3-8b (billed as Workers AI tokens); DLP in the Firewall. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#cloudflare-ai-gateway-managed-edge-free-core]
- Limits (2025): logs free 100K/mo, paid 1M/mo, ~10M/gateway cap; Workers Paid starts $5/mo. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#cloudflare-ai-gateway-managed-edge-free-core]
Kong AI Gateway — plugins on Kong Gateway
- Identity: plugin set on Kong Gateway (self-hosted or Konnect SaaS). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#kong-ai-gateway-plugins-on-kong-gateway]
- API: ai-proxy/ai-proxy-advanced normalize providers to llm/v1/chat. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#kong-ai-gateway-plugins-on-kong-gateway]
- Control plane: AI Semantic Cache (3.8, Redis/Postgres); 6 LLM LB algorithms (3.8); AI (Semantic) Prompt Guard allow/deny -> 4xx; 3.10 RAG injection + PII sanitization; full enterprise API-gateway authn/rate-limit/RBAC. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#kong-ai-gateway-plugins-on-kong-gateway]
Helicone — OSS (Rust), observability-first that also proxies
- Identity: Rust, GPL-3.0; single endpoint, 100+ providers; 0% markup with credits. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#helicone-oss-rust-observability-first-that-also-proxies]
- Control plane: smart routing (model-latency, P2C+PeakEWMA, weighted, cost); rate limits per user/team/global (requests/tokens/dollars); caching on Workers KV (default 7d, max 365d) or self-host Redis/S3; sessions, prompt mgmt, LLM security. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#helicone-oss-rust-observability-first-that-also-proxies]
- Self-host: Web(3000) + Jawn API/proxy(8585) + Worker + Postgres + ClickHouse + MinIO/S3 + Redis. WARNING: port 8585 has no auth by default. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#helicone-oss-rust-observability-first-that-also-proxies]
TrueFoundry — enterprise gateway, low-latency
- Identity: sub-3ms internal latency at enterprise scale. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#truefoundry-enterprise-gateway-low-latency]
- Control plane: rate limiting + token budgeting, per-user/app/tool quotas; guardrails (PII, toxicity); RBAC + per-team keys; MCP gateway (OAuth2/ RBAC/metadata per tool call). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#truefoundry-enterprise-gateway-low-latency]
- Deploy: SaaS, hybrid, or self-hosted/on-prem/air-gapped/multi-cloud; stateless gateway pods + NATS + Postgres + ClickHouse. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#truefoundry-enterprise-gateway-low-latency]
OpenRouter — marketplace/aggregator used as a gateway
- Identity: 315+ models, OpenAI-compatible (base https://openrouter.ai/api/v1). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#openrouter-marketplaceaggregator-used-as-a-gateway]
- Control plane: automatic fallback across providers/models (pay only for successful runs); thinner governance/observability than dedicated gateways. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#openrouter-marketplaceaggregator-used-as-a-gateway]
- Pricing (mid-2026, per a review summary - verify on the pricing page): catalog matches provider; PAYG ~5.5% fee; BYOK first 1M req/mo free, then 5%; free models with ~20 RPM/200-per-day limits. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#openrouter-marketplaceaggregator-used-as-a-gateway]
Vercel AI Gateway — managed, GA
- Identity: unified HTTP API to hundreds of models, one key; GA (2025). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#vercel-ai-gateway-managed-ga]
- Control plane: no markup (pay provider price); BYOK zero fee; $5/mo free credits; automatic retry to other providers; Custom Reporting API (beta) - cost/token/request by model/provider/user-ID/tag/credential, incl. BYOK. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#vercel-ai-gateway-managed-ga]
- Plans (2025/26): Hobby $0 / Pro $20 / Enterprise custom; tight AI-SDK integration. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#vercel-ai-gateway-managed-ga]
AWS Bedrock gateway — a *pattern*, not a product
- Bedrock is not OpenAI-compatible itself. Patterns: [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#aws-bedrock-gateway-a-pattern-not-a-product]
- Bedrock Access Gateway (aws-samples/bedrock-access-gateway) - OSS OpenAI-compat shim; deploy API Gateway + Lambda (Lambda Web Adapter for SSE, no VPC, <=10-min timeout) or ALB + Fargate (lowest streaming latency, no cold starts); Application Inference Profiles for cost tracking; prompt caching (Claude/Nova, up to 90% cost / 85% latency). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#aws-bedrock-gateway-a-pattern-not-a-product]
- LiteLLM/Portkey in front of Bedrock for full multi-provider governance. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#aws-bedrock-gateway-a-pattern-not-a-product]
Databricks AI Gateway — evolving rebrand (CONTESTED naming/GA)
- Lineage: MLflow AI Gateway / MLflow Deployments Server -> Mosaic AI Gateway -> Unity AI Gateway. Primary doc (updated May 2026) says Unity AI Gateway; Data+AI Summit 2025 blog says "Mosaic AI Gateway"; another blog claims "Agent Bricks AI Gateway is GA" - sources disagree; treat GA as unsettled. New LLM/agent/MCP surface is Beta as of May 2026 (no charges during Beta). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#databricks-ai-gateway-evolving-rebrand-contested-namingga]
- Governs: LLM endpoints, agents, MCP servers, coding agents (Cursor / Claude Code / Codex CLI / Gemini CLI). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#databricks-ai-gateway-evolving-rebrand-contested-namingga]
- Control plane: usage tracking, payload logging to Unity Catalog inference tables, per-user/group rate limits, guardrails (PII detection, safety/content filtering), traffic splitting, cost via billable-usage system tables. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#databricks-ai-gateway-evolving-rebrand-contested-namingga]
- Deploy: part of the Databricks lakehouse platform. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#databricks-ai-gateway-evolving-rebrand-contested-namingga]
Selection / decision guidance
- Self-hosted, full control, OSS, data never leaves your VPC -> LiteLLM (broadest provider + key-vault + budget surface) or Portkey OSS (config-driven routing + guardrails). Helicone if observability is the primary need. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#selection-decision-guidance]
- Zero-ops managed, edge latency, free to start -> Cloudflare AI Gateway (caching + dynamic routing + DLP) or Vercel AI Gateway (no-markup + AI-SDK apps). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#selection-decision-guidance]
- Already on an enterprise API gateway -> Kong AI Gateway (reuse Kong RBAC/ authn/rate-limit; add AI plugins). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#selection-decision-guidance]
- Enterprise governance, hybrid/air-gapped, lowest latency, MCP governance -> TrueFoundry or Portkey enterprise. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#selection-decision-guidance]
- All-in on a cloud/data platform -> AWS Bedrock Access Gateway (AWS-native, serverless) or Databricks Unity AI Gateway (lakehouse-native, Unity Catalog audit) - accept the platform lock-in for native cost/audit tables. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#selection-decision-guidance]
- Cheapest path to many models, light governance -> OpenRouter (accept the ~5.5% fee and thinner controls). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#selection-decision-guidance]
- Decision axes: OSS vs managed; data residency (does prompt data leave your network?); markup model (flat fee vs no-markup vs infra-only); native key-vault vs BYOK; guardrail depth (deterministic vs LLM-judge vs PII redaction); audit/RBAC maturity; streaming fidelity; added latency hop tolerance. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#selection-decision-guidance]
Integration patterns
- The OpenAI-compatible drop-in (the universal move - point the SDK at the gateway): [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#integration-patterns]
- LiteLLM proxy config (virtual-key budget + fallback + cooldown): [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#integration-patterns]
- Cloudflare cache + rate-limit headers (per-request override): [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#integration-patterns]
- Portkey config (fallback strategy + input guardrail): [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#integration-patterns]
Anti-patterns & failure modes
- Unauthenticated self-hosted proxy. Helicone's self-host port 8585 has no auth by default - anyone with network access can proxy through it and burn your spend. Firewall or add auth before exposing any self-hosted gateway. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#anti-patterns-failure-modes]
- Semantic-cache false hits. Two prompts with similar embeddings but different intent return the same cached answer - a correctness bug, not just a stale-cache bug. Threshold tuning is the real defense -> llm-routing-cascades. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#anti-patterns-failure-modes]
- Volatile / racing cache. Cloudflare cache is volatile: simultaneous identical requests can both miss. Don't assume a write-then-read is atomic. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#anti-patterns-failure-modes]
- Fallback chains masking degradation. Silent fallback to a weaker model keeps the service "up" while answer quality quietly drops - alert on fallback rate, not just error rate. And retries can amplify load against an already-rate-limited provider (retry storms) - use cooldowns + jitter. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#anti-patterns-failure-modes]
- Gateway as a single point of failure + latency hop. Every request now traverses one component; an outage there takes down all providers at once. The flip side of TrueFoundry's sub-3ms pitch - measure the added hop and run the gateway HA. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#anti-patterns-failure-modes]
- Cost-attribution blind spots under BYOK. When the provider bills you directly (BYOK), the gateway may not see true spend - reconcile gateway spend tables against provider invoices, and prefer gateways that report BYOK traffic (e.g., Vercel Custom Reporting). [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#anti-patterns-failure-modes]
- Leaking raw provider keys. Distributing the real OPENAI_API_KEY to every app defeats the gateway - issue virtual keys and keep provider keys in the vault. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#anti-patterns-failure-modes]
- Treating the gateway as a model router. Reliability fallback != cost/quality model selection; don't hand-roll routing logic in the proxy when the discipline lives in llm-routing-cascades. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#anti-patterns-failure-modes]
2025-2026 frontier
- OpenAI-compat surface is widening (unevenly). Chat Completions is the universal contract; Responses API support is emerging and inconsistent across gateways; several add Anthropic /v1/messages and MCP passthrough (LiteLLM, TrueFoundry, Databricks). Verify per-vendor before assuming Responses works. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#2025-2026-frontier]
- The gateway is becoming the agent/MCP control plane. Databricks Unity AI Gateway, TrueFoundry, and LiteLLM now govern MCP servers, tool calls, and coding agents (Cursor/Claude Code/Codex) - RBAC and audit applied per tool call, not just per completion. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#2025-2026-frontier]
- Guardrails moving inline + LLM-judge based. Cloudflare runs llama-guard-3-8b inline; Portkey ships 40+ guardrails; Kong added on-the-fly embeddings for semantic prompt-guard and PII sanitization. PII/PHI redaction at the proxy is now table-stakes for regulated workloads. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#2025-2026-frontier]
- Pricing models are bifurcating: no-markup / infra-only (Vercel, Helicone OSS, Cloudflare core-free) vs flat platform fee (OpenRouter ~5.5%). BYOK with zero or low fee is the competitive wedge. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#2025-2026-frontier]
- Edge + Rust for latency. Helicone (Rust) and Cloudflare (edge Workers) chase the "gateway adds no latency" promise; sub-3ms internal overhead (TrueFoundry) is now a marketed differentiator. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#2025-2026-frontier]
- Platform-native governance via system tables. Databricks (billable-usage + inference tables in Unity Catalog) and AWS (Application Inference Profiles) fold cost attribution and audit into the data/cloud platform itself. [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#2025-2026-frontier]
Sources
- LiteLLM - AI Gateway (LLM Proxy) overview: https://docs.litellm.ai/docs/simple_proxy [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#sources]
- LiteLLM - Virtual Keys / Budgets / Rate Limits / Fallbacks / Load Balancing / Secret Managers: https://docs.litellm.ai/docs/proxy/virtual_keys , /docs/proxy/users , /docs/proxy/reliability , /docs/proxy/load_balancing , /docs/secret [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#sources]
- Portkey - AI Gateway product + Configs + feature comparison + OSS gateway (npm): https://docs.portkey.ai/docs/product/ai-gateway , https://www.npmjs.com/package/@portkey-ai/gateway [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#sources]
- Cloudflare AI Gateway - Overview / Features / Caching / Rate limiting / Dynamic routing: https://developers.cloudflare.com/ai-gateway/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#sources]
- Cloudflare AI Gateway - Pricing & Limits: https://developers.cloudflare.com/ai-gateway/reference/pricing/ , /reference/limits/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#sources]
- Kong AI Gateway - docs + 3.8 + 3.10: https://developer.konghq.com/ai-gateway/ , https://konghq.com/blog/product-releases/ai-gateway-3-8 , https://konghq.com/blog/product-releases/ai-gateway-3-10 [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#sources]
- Kong - API Gateway vs AI Gateway: https://konghq.com/blog/learning-center/api-gateway-vs--ai-gateway [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#sources]
- Helicone - AI Gateway overview + caching + self-hosting + GitHub: https://docs.helicone.ai/gateway/overview , https://docs.helicone.ai/features/advanced-usage/caching , https://github.com/helicone/ai-gateway [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#sources]
- TrueFoundry - AI Gateway product + on-prem guide: https://www.truefoundry.com/ai-gateway , https://www.truefoundry.com/blog/ai-gateway-on-premise [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#sources]
- OpenRouter - pricing + provider routing + FAQ: https://openrouter.ai/pricing , https://openrouter.ai/docs/guides/routing/provider-selection , https://openrouter.ai/docs/faq [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#sources]
- Vercel AI Gateway - docs + pricing + BYOK + GA changelog: https://vercel.com/docs/ai-gateway , /docs/ai-gateway/pricing , https://vercel.com/changelog/ai-gateway-is-now-generally-available [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#sources]
- AWS Bedrock Access Gateway: https://github.com/aws-samples/bedrock-access-gateway [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#sources]
- Databricks Unity AI Gateway (Beta May 2026; lineage Mosaic/MLflow): https://docs.databricks.com/aws/en/ai-gateway/ , https://www.databricks.com/product/artificial-intelligence/ai-gateway [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#sources]
- Category definition / why AI gateways emerged: https://atlan.com/know/what-is-ai-gateway-llm-gateway/ [archived source: https://llms-explorer.com/sources/mdb-context-hub/llm-ai-gateways/#sources]
Where this helps
- Fronting multiple LLM providers behind one OpenAI-compatible endpoint so applications don't hardcode a single vendor's SDK and can fail over between providers. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Issuing per-team or per-app virtual keys with budgets and rate limits instead of distributing raw provider API keys that can't be revoked or capped individually. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Adding caching, PII/guardrail enforcement, and centralized spend tracking at the proxy layer instead of reimplementing them in every application that calls an LLM. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Deciding between a self-hosted OSS gateway (LiteLLM, Portkey OSS, Helicone), a managed edge gateway (Cloudflare, Vercel), or a platform-native gateway (AWS Bedrock Access Gateway, Databricks Unity AI Gateway) based on data-residency and ops-control requirements. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Project ideas
- Stand up a self-hosted LiteLLM proxy in front of two or more providers, issue virtual keys with per-team budgets, and configure a fallback chain so a provider outage degrades to a backup model instead of failing requests. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Build a Cloudflare AI Gateway in front of an app's LLM calls and configure exact-match caching plus dynamic routing (conditional/percentage/budget nodes) to cut duplicate-request cost. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Implement cost-attribution reconciliation for a BYOK gateway setup — compare the gateway's reported spend against actual provider invoices to catch the blind spots BYOK billing creates. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Prototype an MCP-governance layer on top of an existing gateway (LiteLLM or TrueFoundry) that applies RBAC and audit logging per tool call for a fleet of coding agents. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Common mistakes
- Exposing a self-hosted gateway's admin or proxy port without authentication — Helicone's self-host port 8585 has no auth by default, letting anyone with network access proxy through it and burn spend. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Treating the gateway as a model router for cost/quality selection rather than a reliability layer — that routing-and-cascade logic belongs to a dedicated routing discipline, not hand-rolled proxy config. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Assuming a write-then-read against the gateway cache is atomic — Cloudflare's cache is volatile, so simultaneous identical requests can both miss and both hit the upstream provider. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Letting silent fallback chains mask quality degradation — a fallback to a weaker model keeps the service technically 'up' while answer quality drops without anyone noticing, unless fallback rate itself is alerted on. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Known issues
- Semantic caching (embedding-similarity hits across differently-worded prompts) can produce false hits — two prompts with similar embeddings but different intent return the same cached answer, which is a correctness bug, not just staleness. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- A gateway is a new single point of failure and adds a latency hop in front of every provider call — an outage at the gateway takes every provider down at once, even the ones that are individually healthy. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Pricing, feature sets, and even product naming shift fast in this space (e.g. Databricks' Mosaic/Unity AI Gateway rebrand) — anything quoted here should be re-verified against the vendor's current docs before being relied on. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- OpenAI-compatible surface coverage is uneven across gateways — Chat Completions is near-universal, but Responses API and Anthropic /v1/messages passthrough support varies and can silently limit which app patterns a given gateway supports. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Children
- Unified OpenAI-compatible API surface (frontier)
- Virtual keys & key-vault management (BYOK) (frontier)
- Rate limiting & quotas (frontier)
- Budget/spend controls & cost attribution/chargeback (frontier)
- Fallback chains, retries & load balancing (frontier)
- Gateway caching (exact-match vs semantic) (frontier)
- Guardrail & PII/DLP enforcement at the proxy (frontier)
- Observability, logging & tracing hooks (frontier)
- Governance, RBAC & audit (frontier)
- Self-hosted vs managed vs edge vs platform-native (frontier)
- Streaming (SSE) pass-through (frontier)
- MCP/agent/tool-call gateway governance (frontier)
- LiteLLM gateway and SDK engineering
Frontier under this node: Budget/spend controls & cost attribution/chargeback, Fallback chains, retries & load balancing, Gateway caching (exact-match vs semantic), Governance, RBAC & audit, Guardrail & PII/DLP enforcement at the proxy, MCP/agent/tool-call gateway governance, Observability, logging & tracing hooks, Rate limiting & quotas, Self-hosted vs managed vs edge vs platform-native, Streaming (SSE) pass-through, Unified OpenAI-compatible API surface, Virtual keys & key-vault management (BYOK)