Blank white background with no objects or features visible.

Hello, EMEA! TrueFoundry is growing our footprint across Europe, the Middle East, and Africa. Read more. →

What Is an LLM Gateway? How It Works, What to Look For, and When You Need One

By TrueFoundry

Updated: November 16, 2026

Published: August 8, 2024

 What is an LLM Gateway
⚡ TL;DR
  • An LLM gateway sits between your apps and every model provider, and handles routing, auth, budgets, guardrails, failover and logging in one place.
  • You'll want one once more than one team, model or production app is involved.
  • In 2026 it's increasingly governing MCP tools and agents too, not just model calls.
  • Below: how it works, what to check when choosing one, and how the main options compare.

Ask five platform engineers what an LLM gateway is and you'll get five slightly different answers. The short version: it's a layer that sits between your apps and the model providers they call, whether that's OpenAI, Anthropic, Google, Bedrock or something you host yourself. Every request goes through it. So instead of each team wiring up its own SDKs, keys and retry logic, you get one API, one place to set budgets and guardrails, and one log that tells you who called which model and what it cost.

Most teams don't go looking for a gateway. They end up needing one, usually after a rough week. A provider goes down for an hour and takes the support bot with it. Someone's test script loops over a weekend and quietly eats a month of budget. Or security asks for a list of every app sending customer data to an LLM, and nobody can answer.

All three of those problems happen in the same place: the request path. That's exactly where a gateway sits.

Why do teams need an LLM gateway?

Calling a provider directly is fine when you have one app. It gets messy around the third or fourth, and a few things tend to break first.

Every provider speaks its own dialect. OpenAI, Anthropic, Bedrock and a vLLM server all format requests, errors and streaming a little differently, so each app ends up carrying its own adapter code.

Then there are the keys. They live in env files, CI secrets, a notebook someone forgot about. Rotating one turns into a scavenger hunt.

Cost is the one finance notices. The invoice is a single number. It won't tell you which team or feature spent it, and by the time it arrives, the money's already gone.

Outages hurt too. If your code calls one model directly, that model's bad hour is your bad hour, unless every team writes its own retry and fallback logic. Most don't.

And in a regulated industry, sooner or later someone will ask you to show what data went to which model. Logs scattered across ten services won't get you there.

You can't really fix any of this inside a single app. These are shared problems, so they need a shared layer.

Spending more on LLMs than you can explain?

TrueFoundry attributes every token to a team, app or user, and enforces budgets before the spend happens, not after the invoice arrives.

See it on your own traffic → Book a 20-min demo

How does an LLM gateway work?

Under the hood, the flow is fairly simple. A request comes in, gets checked, gets routed to a model, and gets logged on the way out. In a bit more detail:

  1. Your app sends the request to the gateway instead of straight to the provider.
  2. The gateway checks who's calling and whether they're allowed to, then applies rate limits and budgets. If a team is over budget, the request stops right here, before it costs anything.
  3. Input guardrails run next, such as PII masking, prompt-injection checks or your own rules.
  4. The LLM router picks a model based on weights, latency, cost or how hard the task looks. If that model errors out, it tries the next one.
  5. The gateway translates the request into that provider's format, then turns the response back into the shape your app expects.
  6. Tokens, latency, cost and the full trace get recorded against whoever made the call.

For developers, switching over is usually a one-line change. Most gateways speak the OpenAI API, so you keep your existing SDK, point it at a different base URL, and pick models by name:

from openai import OpenAI

# Point the SDK at the gateway, not the provider
client = OpenAI(
    base_url="{GATEWAY_BASE_URL}",
    api_key="your_truefoundry_api_key",  # one key per team or app
)

# Switch providers by changing the model string
response = client.chat.completions.create(
    model="provider-account/model-name",
    messages=[{"role": "user", "content": "Summarize this ticket"}],
)

What about latency? A good gateway adds a few milliseconds, which disappears next to a model response that takes hundreds. In our own benchmark, TrueFoundry's gateway added about 3–4 ms and handled 350 requests per second on a single CPU core (full numbers here).

What features should an LLM gateway have in 2026?

If you're comparing gateways, these are the things we'd check first. The first five matter for almost every production team; the rest depend on your setup.

FeatureWhat it doesMust-have?
Unified APIOne OpenAI-compatible endpoint for every provider and self-hosted modelYes
Routing + fallbacksRoutes by weight, latency, cost or complexity; fails over automatically when a provider errorsYes
Budgets + rate limitsHard caps per team, app or user, enforced before the request runsYes
GuardrailsPII masking, prompt-injection defense, content moderation, custom policiesYes for regulated teams
Observability + cost attributionFull traces, token and cost per request, export to your stack via OpenTelemetryYes
Identity + accessSSO through your IdP, SCIM, role-based access, scoped keysYes for enterprises
Deployment + data residencyRuns in your VPC or on-prem, keeps logs in your regionDepends on compliance needs
CachingExact and semantic caching to cut repeat costsNice to have
MCP + agent governanceControls which tools and agents can be called, and by whomYes if you run agents

LLM gateway vs API gateway vs LLM proxy vs LLM router

People use these terms interchangeably, which causes real confusion on vendor calls. Roughly speaking, a proxy forwards traffic, a router chooses the model, an API gateway manages REST APIs, and an LLM gateway does the first two plus token-level control.

LLM proxyLLM routerAPI gateway (Kong, Apigee)LLM gateway
Main jobForward requests to a providerChoose the best model per requestManage REST/GraphQL APIsGovern all LLM traffic
Understands tokens and costBasicPartlyNoYes
Fallbacks across providersRarelyYesNoYes
Guardrails + PIINoNoThrough pluginsBuilt in
Budgets per teamNoNoRequest-count limits onlyYes, in tokens and dollars
Governs MCP tools + agentsNoNoNoYes (modern gateways)

Want more depth? We've written separately about LLM proxies, AI gateways vs API gateways and MCP gateways vs proxies vs routers.

Do you actually need an LLM gateway?

Not always, and it's worth saying that plainly. One app, one provider, a small bill? Call the provider directly and come back to this later. A gateway is one more piece of infrastructure, and it should earn its place.

It's probably time if a couple of these sound familiar:

  • More than one team is building with LLMs
  • You use, or want to try, more than one model provider
  • Finance asked which team is driving the LLM bill and nobody could say
  • A provider outage has already hit a customer-facing feature
  • Security or compliance wants an audit log of prompts and responses
  • Developers are pointing coding agents like Claude Code or Cursor at company systems
Direct API callsLLM gateway
SetupFast for the first appOne integration, then every app reuses it
Switching modelsCode change in every appChange a config value
OutagesYour app goes down with the providerAutomatic fallback to another model
Cost visibilityOne invoice, no breakdownPer team, app and user, in real time
SecurityKeys and policies managed per appCentral keys, guardrails and audit trail

What does this look like at scale?

A Fortune 50 healthcare company runs an AI agent handling 500 million IVR calls on TrueFoundry. FloQast governed 4M+ AI requests across 6 model providers in 90 days, with a 98.9% success rate.

Get a live walkthrough →Read the case studies

From LLM gateway to AI gateway: MCP tools and agents

Here's what changed over the last year or so. Agents don't just call models anymore. They call tools, usually through the Model Context Protocol (MCP), and those tools are your CRM, your database, your ticketing system. Each of those calls needs the same treatment a model call gets: who's allowed, with which permissions, and a record of what happened.

So the category is stretching. When vendors say "AI gateway" now, they usually mean one control plane for three kinds of traffic: model calls, tool calls through an MCP gateway, and the agents themselves through an agent gateway, which handles agent identity, a shared registry and human approval for risky actions.

You can see the shift in the market, too. Gartner is putting out its first Magic Quadrant for AI gateways this year, and in April Palo Alto Networks said it would buy Portkey (our take on that deal). If you're choosing a gateway today, make sure it can grow into MCP and agent governance. Otherwise you'll probably be shopping again within a year.

Popular LLM gateway options in 2026

Which one fits depends on what you care about most: running open source yourself, enterprise governance, or quick access to lots of models. Here's how we see the main options, trade-offs included:

OptionTypeBest forWatch out for
TrueFoundry AI GatewayLLM + MCP + agent gateway; runs in your VPC or on-premEnterprises that need governance, data residency and agent controlMore than a solo developer needs
LiteLLMOpen-source proxyTeams happy to self-host and maintain itYou own patching, scaling and security; its PyPI package was hit by a supply-chain attack in March 2026
PortkeyManaged gatewayTeams wanting a hosted gateway with observabilityPalo Alto Networks plans to acquire it, which may shift its roadmap toward security
Kong AI GatewayAPI gateway with AI pluginsTeams already running Kong for APIsBuilt for REST first; token-level budgets and MCP governance are add-ons
OpenRouterHosted model marketplaceFast access to hundreds of models with one keyTraffic runs through a third party; limited team-level governance
Cloudflare AI GatewayEdge proxyCaching and basic analytics for web appsLight on guardrails, budgets and self-hosting

For a longer comparison covering latency, pricing and enterprise features, see our best LLM gateways guide.

Best practices for rolling out an LLM gateway

A few lessons that save a lot of pain later.

Start with whichever app is noisiest, the one with the biggest bill or the most incidents. A visible win there makes it far easier to bring other teams along.

Once apps have moved over, revoke the direct provider keys. A gateway people can route around doesn't govern anything.

Put budgets in on day one, for every team and environment. Adding limits after people are used to having none is a much harder conversation.

Give every production model a fallback from a different provider, and actually test the failover before you need it.

Finally, send traces to the tools your team already watches, like Datadog or Grafana, through OpenTelemetry instead of adding yet another dashboard.

The bottom line

An LLM gateway turns a pile of scattered model calls into something you can actually govern. If you run one small app, you can wait. Once several teams are shipping LLM features, or agents start touching company systems, the gateway is where cost control, reliability and security really get enforced. Just pick one that handles MCP tools and agents as well, because that's where the traffic is heading.

3-week pilot in your cloud

One control plane for every model, tool and agent

We deploy TrueFoundry in your own cloud and prove it on 2–3 of your real workloads: cost, latency and reliability against your current setup.

Frequently asked questions

What is the difference between an LLM gateway and an AI gateway?

An LLM gateway handles calls to language models. An AI gateway covers those plus MCP tool calls and AI agents, all from one control plane. In practice, most LLM gateways are turning into AI gateways as agent traffic grows.

Is an LLM gateway the same as an LLM proxy?

Not quite. A proxy mostly forwards requests to a provider. A gateway adds routing, fallbacks, budgets, guardrails, access control and full observability on top of that.

Does an LLM gateway add latency?

A little, yes. A well-built gateway adds a few milliseconds per request, which is tiny compared with model response times. TrueFoundry's adds about 3–4 ms.

Can I self-host an LLM gateway?

Yes. You can self-host open-source options like LiteLLM, though you'll be the one maintaining and securing them. Enterprise gateways like TrueFoundry can also run entirely in your own VPC or on-prem, so prompts and logs never leave your environment.

How does an LLM gateway reduce costs?

Mainly three ways: it sends simple requests to cheaper models, caches repeated answers, and stops overspending before it happens through budgets. It also shows cost per team and app, so you finally know where the money goes.

What makes an LLM gateway secure?

Look for SSO through your identity provider, role-based access, scoped keys, PII masking, prompt-injection guardrails, a full audit trail, and the option to run it inside your own network. Our guide to what security teams need from an AI gateway goes deeper.

Try now.

One gateway for all your models, MCP servers, and agents. No credit card needed.

Start free
Table of Contents

One Gateway for Every LLM, Agent and MCP Server

Book a 30-min with our AI expert

Book a Demo

The fastest way to build, govern and scale your AI

Book Demo
Summarize with
ChatGPT logo by OpenAI
Perplexity AI logo
Blurry red snowflake on white background, symmetrical frosty design with soft edges and abstract shape.

Discover More

November 5, 2025
|
5 min read

Data Residency in the Age of Agentic AI: How AI Gateways Enable Sovereign Scale and Compliance

August 27, 2025
|
5 min read

Mapping the On-Prem AI Market: From Chips to Control Planes

August 27, 2025
|
5 min read

AI Gateways: From Outage Panic to Enterprise Backbone

Secure AI Gateway with MCP: Enterprise-Ready Protection
July 4, 2025
|
5 min read

Secure AI Gateway with Centralized MCP for Enterprises

November 18, 2026
|
5 min read

Claude Code Limits Explained: Pro, Max and Ultra

No items found.
Analyzing how Cloudflare detects MCP traffic
November 18, 2026
|
5 min read

How Cloudflare Detects MCP Traffic and Helps Secure It

No items found.
November 18, 2026
|
5 min read

What Is AI Search? How It Works, Architecture, and Enterprise Use Cases

No items found.
Explaining LLM fine-tuning for enterprise AI teams
November 18, 2026
|
5 min read

A Practical Guide to LLM Fine-Tuning: Methods, Data, Costs, and Deployment

No items found.
No items found.

Recent Blogs

Black left pointing arrow symbol on white background, directional indicator.
Black left pointing arrow symbol on white background, directional indicator.
Take a quick product tour
Start Product Tour
Product Tour