# LLM Gateway

One door to every model provider, with budgets, guards and per-call cost recorded on the way through.

Source: https://docs.omazy.ai/reference/llm-gateway/

Every answer your agent gives costs money and takes time, and both are easy to
discover only at the end of the month. The gateway is the single door all model
traffic goes through, which is what makes those two numbers visible while you
can still do something about them.

Calling a provider directly is a bit like giving everyone in the building their
own front door key. It works right up until you want to know who came in.

## What it does

| Surface | Answers |
|---|---|
| **Providers** | Which vendors and models are available, and whose keys pay for them. |
| **Budgets** | How much may be spent, over what period, by whom. |
| **Guards** | Which models are permitted, and what a request may contain. |
| **Usage** | How many calls, which models, how many tokens. |
| **Spend** | What it actually cost. |
| **Forecast** | What it is on track to cost. |

## Bring your own keys

You can run on the platform's pooled capacity or supply your own provider
credentials. Own keys mean your own rate limits and your own bill, which is
usually what a larger workspace wants once traffic is predictable.

Credentials are stored encrypted and are never returned by the API after they
are set. If you need to know whether a key still works, the gateway will tell
you it is healthy. It will not tell you what it is, and neither will anybody
else, which is the entire point.

## Budgets and guards are different tools

They get conflated, so it is worth separating them plainly.

A **budget** is about money. It caps spend over a period and it does not care
which model burned it.

A **guard** is about permission. It decides whether a particular request is
allowed at all: which models may be used, and what may be sent to them.

A budget stops you spending too much on the right thing. A guard stops you doing
the wrong thing cheaply. You want both.

## When a provider has a bad day

Upstream rate limits do not always arrive wearing a name badge. A gateway can
return a generic unavailable error when what actually happened is that your
provider account hit its tokens-per-minute ceiling. If calls start failing in
bursts and recovering on their own, suspect your provider's throughput limit
before you suspect the model.

## Coming in this section

Configuring providers and keys, setting budgets, writing guards, and reading
usage against spend.