> ## Documentation Index
> Fetch the complete documentation index at: https://docs.infery.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Rate limits & retries

> Survive 429s with backoff, queueing, and fallbacks — without hammering the gateway.

For the actual limits per plan, see [Rate limits](/api-reference/rate-limits). This guide is the playbook for handling them in real applications.

## The two failure modes

| Code                       | Meaning                                                  | Retry?                     |
| -------------------------- | -------------------------------------------------------- | -------------------------- |
| `429 rate_limit_exceeded`  | Your key burst above its RPM, or hit the daily token cap | Yes — honour `Retry-After` |
| `502 / 503 / 504`          | Upstream provider transient                              | Yes — exponential backoff  |
| `400 / 401 / 403 / 422`    | Caller error                                             | No — fix the request       |
| `402 insufficient_credits` | Out of balance                                           | No — top up                |

Don't blanket-retry. Retrying caller errors burns RPM for nothing.

## Always honour `Retry-After`

Every 429 from us includes:

```
HTTP 429 Too Many Requests
Retry-After: 12
```

The number is **seconds until your window frees**. Use it directly — don't override with your own delay.

```python python theme={null}
import time

def call_with_429(req):
    while True:
        resp = req()
        if resp.status_code != 429:
            return resp
        wait = float(resp.headers.get("Retry-After", "5"))
        time.sleep(wait)
```

## Exponential backoff with jitter (for 5xx)

For upstream transients without `Retry-After`:

```python python theme={null}
import random, time

def with_backoff(req, max_attempts=5):
    for attempt in range(max_attempts):
        resp = req()
        if resp.status_code < 500 and resp.status_code != 429:
            return resp
        if resp.status_code == 429:
            time.sleep(float(resp.headers.get("Retry-After", "5")))
            continue
        # 5xx
        delay = (2 ** attempt) + random.random() * 0.5
        time.sleep(delay)
    return resp  # last attempt's response, even if failed
```

Jitter (`+ random()`) is critical — without it, every client retries at the same instant and you get a stampede.

## Don't retry 4xx

`400/401/403/422` are deterministic — retrying just wastes RPM and money. Fix the request, then resubmit. Common offenders:

* Wrong model slug → check `GET /v1/models`
* Missing required parameter → check the endpoint reference
* Image too large → resize before resending
* Malformed JSON → fix the producer

## SDK-level retries

The OpenAI SDK retries 429s and 5xx automatically:

```python python theme={null}
client = OpenAI(
    api_key=API_KEY,
    base_url="https://api.infery.ai/v1",
    max_retries=4,           # default 2
    timeout=60.0,
)
```

This honours `Retry-After`. For most apps, this is enough — you don't need a custom loop. **But:**

* It retries on a single call. Bursty workloads still need a queue.
* It applies to streaming too — the first chunk is what matters.

## Stay under the limit on purpose

Reactive retry is the floor. Proactive limiting is the ceiling. Token-bucket on your side, capped at \~80% of the key's RPM:

```python python theme={null}
import threading, time, collections

class TokenBucket:
    def __init__(self, rate_per_min: int):
        self.capacity = rate_per_min
        self.tokens = float(rate_per_min)
        self.refill_per_sec = rate_per_min / 60.0
        self.last = time.monotonic()
        self.lock = threading.Lock()

    def take(self, n: int = 1):
        while True:
            with self.lock:
                now = time.monotonic()
                self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.refill_per_sec)
                self.last = now
                if self.tokens >= n:
                    self.tokens -= n
                    return
            time.sleep(0.05)

bucket = TokenBucket(rate_per_min=int(120 * 0.8))  # 80% of a 120 rpm key

def safe_call(req):
    bucket.take(1)
    return req()
```

Result: zero 429s under steady load. Bursts above 120 rpm spike → bucket pauses → resumes when refilled. Queue any requests that need to go through.

## Fallback chains: the better answer for production

Per-call retries help, but [fallback chains](/workspaces/fallback-chains) help more. Configure once:

```
Source: gpt-4o
└─ Fallback 1: gpt-4o-mini
└─ Fallback 2: gemini-2-5-flash
```

Now a 429 on `gpt-4o` is invisible to your code — the gateway routes to `gpt-4o-mini`, and you see:

```
x-model-used: gpt-4o-mini
x-fallback-from: gpt-4o
x-fallback-depth: 1
```

This shifts retry from your client to the gateway, with sub-100 ms hop instead of an exponential wait. Combine with exponential backoff for the case when **all** fallbacks are also rate-limited.

## Per-environment keys

Don't share one key across dev/staging/prod. Reasons:

* Dev experiments shouldn't drain prod's RPM budget
* Leaked dev keys have lower blast radius if scoped to a low-RPM preset
* Per-env analytics are clearer

Settings → API Keys → Create. Pick a [quota preset](/workspaces/quotas-and-presets) per environment.

## Daily token caps

Some plans cap **total tokens per day** in addition to RPM. Hitting the cap returns 429 too — but `Retry-After` will be **the seconds until midnight UTC**, not seconds. Don't blindly sleep — instead:

* Reduce request volume
* Switch to a cheaper model
* Top up to a higher plan

You'll see `code: "rate_limit_exceeded"` and `message: "Daily token cap reached"` to distinguish from RPM 429s.

## When backoff fails

If you back off twice and still 429:

1. Look at **Settings → Usage → By key** — is one key dominating?
2. Check **Settings → API Keys → preset** — is the preset lower than you remember?
3. Did you ship a loop without rate limiting? Look at request volume in the last hour.
4. Open a ticket — sometimes it's our problem and we want to know.