# Providers and models

Connecting a model provider, choosing a model, and what happens when one is unavailable.

Source: https://docs.omazy.ai/reference/llm-gateway/providers/

A provider is a vendor account. A model is one thing that vendor will run for
you. Connecting the first does not automatically give you every one of the
second, and assuming otherwise is the most common surprise here.

## Connecting a provider

Console, Workspace, LLM providers. You supply a credential, the gateway probes
it, and the provider comes up healthy or it does not.

The probe uses one nominated model. A healthy provider means that model
answered. It does not certify every model the vendor lists, so treat health as
"the door opened", not "everything inside works".

## The model catalogue

The gateway caches each vendor's advertised model list so the console has
something to show without calling out on every page load. Two consequences worth
knowing:

- A model added by the vendor this morning may not appear in the list yet.
- Vendors will usually serve models that were never in the list at all, so an
  absent name is not proof of an unavailable model.

The catalogue is a convenience, not a contract.

## Choosing a model

The temptation is to pick the largest model available and stop thinking about
it. Resist it, for two reasons that have nothing to do with cost.

Bigger models are slower, and in a live chat a slow correct answer loses to a
fast good one more often than anyone likes to admit. And a model that is
excellent at reasoning is not automatically excellent at following a brief,
which is most of what an agent actually does.

Change the model, then run your agent's tests before publishing. Model choice is
a configuration change with the blast radius of a code change.

## Failure modes

| Symptom | Usual cause |
|---|---|
| Bursts of failures that clear on their own | Provider throughput limit, not the model |
| Every call fails immediately | Credential rejected or revoked |
| One model fails, others fine | That model was deprecated or renamed by the vendor |
| Answers arrive but are truncated | An output token limit is set lower than the answer needs |

That last one deserves a word. An unset maximum is not the same as an unlimited
one. Leave it unset and you inherit a default, and the default is smaller than
you think. Long answers get guillotined mid-sentence, which reads to a customer
as the agent losing its train of thought.