LLM Gateway: Routing, Fallbacks and Rate Limits Across Providers

What a gateway is and why you need one

An LLM gateway is a thin service between your applications and model providers. Instead of each service calling OpenAI, Anthropic or Google directly, they call one internal endpoint with a task name. The gateway decides which model to use, retries or switches provider on failure, checks limits, and records what happened.

You need it as soon as you have more than one AI feature or more than one provider. Without it, API keys spread across repositories, every team writes its own retries, and nobody can answer "how much did feature X cost last week?".

Core responsibilities

Routing by task, not by model name

Applications should request a task profile, not a specific model. The mapping lives in configuration:

routes:
  classify-ticket:
    primary: small-fast-model
    fallback: [other-provider-small]
    max_output_tokens: 50
  draft-reply:
    primary: large-model
    fallback: [other-provider-large, small-fast-model]
    timeout_ms: 20000
  contract-analysis:
    primary: self-hosted-model
    fallback: []          # sensitive data, never leaves our servers

This lets you switch a feature to a cheaper or newer model without redeploying the application. Useful routing signals:

• task type: simple classification goes to a small model, reasoning to a large one;
• input size: long documents go to models with a large context window;
• data sensitivity: personal data goes to a self-hosted or approved model only;
• user tier: paying customers may get a stronger model.

Fallbacks that do not make things worse

Not every error deserves a retry. Separate them:

• Retry the same provider with exponential backoff and jitter: timeouts, 429 rate limits, 5xx errors.
• Switch provider: repeated failures, or a provider marked unhealthy.
• Do not retry: 400 validation errors, content policy refusals, context too long. Fix the request instead.

Add a circuit breaker: after several failures in a short window, stop sending traffic to that provider for a while and probe it periodically. This prevents a slow outage from eating your timeouts.

Two caveats:

• prompts tuned for one model may behave differently on another, so test every fallback with your eval set;
• for streaming responses, failover is only safe before the first token reaches the user.

Limits and budgets

Providers limit you by requests and tokens per minute. Your gateway should limit your users, so one client or one runaway script cannot exhaust the shared quota.

• Rate limits: requests and tokens per minute per user, using a token bucket in Redis or a similar store.
• Budgets: daily or monthly spend per user, team and feature. Estimate cost before the call from input tokens and the output cap, then record the actual cost after.
• Hard and soft limits: at a soft limit, route to a cheaper model; at a hard limit, reject with a clear message.
• Output caps: always set a maximum output length per route.

Keys and logging

Provider keys are stored only in the gateway, in a secrets manager, never in client applications. Applications authenticate to the gateway with their own internal keys, which you can revoke individually.

Log for every request: route, model, provider, user and feature IDs, input and output tokens, cost, latency, status and fallback path. Store prompt and response bodies only where policy allows, with masking of personal data and a short retention period.

Build or adopt

Open-source and managed gateways already cover a unified API, fallbacks and cost tracking. Adopt one if it fits; build your own thin layer when you need custom routing rules, strict data residency or deep integration with your billing. In both cases, keep route configuration in version control.

Common mistakes

• Hardcoding model names in application code.
• Retrying non-retryable errors and multiplying cost.
• Fallback models that were never tested on your prompts.
• Logging full prompts with personal data forever.
• No per-user limits, so one tenant blocks everyone.