AI Gateway & LLM Routing 2026 Guide: Portkey, Kong AI Gateway, LiteLLM, Cloudflare AI Gateway & Helicone Compared

<h2>Market and 2026 trends</h2> <p>The AI Gateway market is projected to grow from $500M in 2024 to $8B by 2030 (45% CAGR). McKinsey's GenAI Productionization survey finds 70% of enterprise GenAI adopters cite runaway LLM cost, multi-provider lock-in, compliance risk, and lack of observability as their top challenges. Deploying an AI Gateway typically delivers LLM cost -50% (semantic cache + smart routing), latency -40%, multi-provider failover uptime 99.95%+, zero PII leaks, full team/project cost visibility, and 100% spend-cap compliance. AI Gateways and LLM routers unify: (1) a universal API spanning 200+ providers (OpenAI, Anthropic, Google, Cohere, Mistral, Bedrock, Vertex); (2) smart routing (task → best model — GPT-4o vs. Sonnet 4.6 vs. Haiku cost/quality optimization); (3) semantic cache (embedding-similar queries → 0 tokens, cost -40%); (4) fallback / retry (auto failover with exponential backoff); (5) rate limits and spend caps per team/user/project; (6) guardrails (PII redaction, prompt-injection blocking, output filters); (7) observability (LangSmith / Langfuse / Helicone — trace / cost / latency); (8) A/B testing for prompts and models; (9) prompt management (versioning, deployment, CI/CD); and (10) audit logs for SOC2 / HIPAA / EU AI Act compliance.</p>

<h2>Leading AI Gateways and LLM routers compared</h2> <ul> <li><strong>Portkey</strong> ($15M, 1,000+ customers — Postman, Springworks, Haptik): all-in-one AI Gateway + Prompt Library + Observability + Guardrails, 200+ providers, semantic cache, $49-$499 / month (Cloud or self-hosted).</li> <li><strong>Kong AI Gateway</strong> (built on Kong's $1.4B platform, 900+ enterprise customers — Verizon, Honeywell, Cisco, Yahoo): Kong Gateway extension with AI proxy + prompt guard + semantic caching + rate limiting, runs alongside the API gateway, $50K-1M+ / year.</li> <li><strong>LiteLLM</strong> (open source, 10,000+ stars, BerriAI / YC, used by Anthropic, Lemonade, Adobe): universal Python SDK and proxy, 100+ providers, free self-hosted + LiteLLM Cloud $99-$999 / month.</li> <li><strong>Cloudflare AI Gateway</strong> (Workers AI native, 100,000+ developers): free tier, analytics + caching + rate limit + logs, $Free-$200 / month (Workers Paid).</li> <li><strong>Helicone</strong> ($2M Seed, YC W23, 2,000+ customers — Mintlify, Cognosys): LLM observability + proxy, cost tracking + caching, Free-$500 / month.</li> <li><strong>OpenRouter</strong> (open source + SaaS, 100,000+ developers): 300+ models on one API, pay-as-you-go.</li> <li><strong>Langfuse</strong> (open source, $4M Seed, 5,000+ customers — Khan Academy, Twilio, Samsara): LLM observability + prompt management + evaluation, Cloud $59-$599 / month or free self-hosted.</li> <li><strong>LangSmith</strong> (LangChain, $25M, 15,000+ customers — Klarna, Elastic, Moody's): LangChain-native tracing + evaluation + annotation, $39 / user / month → enterprise.</li> <li><strong>TrueFoundry</strong> ($19M — Ola, Razorpay, Atlassian): MLOps + LLM gateway + self-hosted LLM, $50K-500K / year.</li> <li><strong>Vellum</strong> ($5M — Faire, Rec Room): prompt engineering + eval + deployment + router, $500-$5K / month.</li> <li><strong>Martian Router / Not Diamond</strong>: AI-native routers that auto-select the best model on the cost-quality Pareto frontier.</li> <li><strong>Lakera, Protect AI, Promptfoo, Braintrust, PromptLayer, W&B Weave, Arize Phoenix</strong>: complementary security / evaluation / observability layers.</li> </ul>

<h2>Recommended stacks by use case</h2> <p>Picks: (A) Startup MVP (single provider) = LiteLLM Proxy + Helicone Free + self-hosted Langfuse, ~$50/mo, gets you token visibility and caching. (B) Mid-stage (OpenAI + Claude) = Portkey or OpenRouter + Langfuse Cloud, ~$500/mo for failover + cost tracking. (C) Production SaaS = Portkey Enterprise + Langfuse + Lakera guardrails, $2-10K/mo for SOC2 + PII redaction. (D) Enterprise on Kong = Kong AI Gateway + LangSmith + Arize, ~$200K/yr for unified API + AI gateway. (E) Cloudflare Workers stack = Cloudflare AI Gateway + Workers AI + Vectorize + R2, $200-$2K/mo edge-AI native. (F) LangChain-native = LangSmith + Portkey + OpenRouter, ~$1K/mo. (G) Cost-critical with self-hosted = TrueFoundry + vLLM + LiteLLM, ~$100K/yr (Llama 3.1 / Mixtral self-hosted + OpenAI fallback). (H) Prompt engineering = Vellum or Braintrust + Portkey, ~$2K/mo for prompt CI/CD. (I) Smart routing = Martian or Not Diamond + Portkey, ~$500/mo auto-best-model. (J) Compliance (finance/health) = self-hosted Portkey + Lakera + SOC2/HIPAA, ~$200K/yr zero PII leaks. (K) Multi-tenant SaaS = Portkey or LiteLLM + Stripe metering, $1-5K/mo with per-customer spend caps. Target KPIs: LLM cost -50%, latency -40%, uptime 99.95%+, zero PII leaks, 100% spend-cap compliance, full per-team cost visibility.</p>

<h2>2026 trends and roadmap</h2> <p>Trends: semantic-cache maturation (embedding hit rate 30-60%, cost -40%, Redis / Pinecone vector cache); smart routing (task classifier → best model, cost -30% while preserving quality); self-hosted LLM hybrid (vLLM + Llama 3.1 70B / Mixtral 8x22B + OpenAI fallback, cost -70%); guardrails standardization (NeMo Guardrails + Lakera + Llama Guard); prompt CI/CD (Vellum / Braintrust / Langfuse — versioning + eval + deploy, regression prevention); multi-provider failover (99.95%+ uptime); OpenTelemetry-native tracing (LangSmith / Langfuse / Phoenix OTel-compatible); token spend caps with Slack alerts; audit logs for SOC2 / HIPAA / EU AI Act; edge AI gateways (Cloudflare / Vercel, latency -50%). Roadmap: Week 1 demo Portkey/Kong/LiteLLM/Helicone/Cloudflare + provider inventory + token spend baseline + compliance requirements; Month 1 AI gateway proxy + multi-provider routing + basic caching + observability dashboard = cost -20%, visibility 100%; Months 2-3 semantic cache + smart routing + guardrails (PII redaction, prompt-injection blocking) + spend caps = cost -40%, zero PII leaks; Month 6 self-hosted hybrid + prompt CI/CD + OpenTelemetry + SOC2/HIPAA audit = cost -60%, latency -30%, compliance complete; Year 1 full deployment (multi-provider + self-hosted + smart router + guardrails + audit) = cost -50%, latency -40%, uptime 99.95%+, zero PII leaks, full team cost visibility.</p>