Find your real throughput — and whether requests-per-minute (RPM) or tokens-per-minute (TPM) is the bottleneck.
Most LLM APIs enforce two ceilings at once: a cap on requests per minute and a cap on tokens per minute. Your real throughput is whichever runs out first. If your requests are token-heavy, TPM bites before RPM; if they're tiny and frequent, RPM bites first. This tool shows the binding limit so you know what to ask support to raise.
The crossover is worth knowing: divide the TPM limit by the RPM limit and you get the average request size at which the two ceilings bind equally. A 500 RPM / 200,000 TPM tier balances at 400 tokens per request. Above that size the token limit decides your throughput and raising RPM changes nothing.
| Counts | Often missed |
|---|---|
| Input tokens — prompt, system prompt, conversation history | History grows every turn, so a chat session's per-request cost rises as it runs |
| Output tokens the model generates | Usually estimated up front from your max-tokens setting, not from what was returned |
| Retries after a failure | A retry is a new request against both ceilings |
| Every parallel worker you run | Limits apply per account or per key, not per process |
The last one causes most surprise 429s. Ten workers each politely staying under the limit will still collectively blow through it, because the ceiling is shared.
Trim the input. Token limits are usually dominated by context, not by output. Dropping unused history or trimming an oversized system prompt raises effective throughput immediately and lowers cost at the same time.
Set max-tokens honestly. Where output is reserved against the limit up front, an inflated max-tokens value spends quota on tokens that are never generated.
Route small work to a smaller model. Limits are per model, so classification and extraction handled by a cheaper model free up the flagship's quota for work that needs it.
Back off properly. On a 429, honour the retry-after header and use exponential backoff with jitter. Retrying immediately turns one rejected request into several and makes the burst worse.
The published number is per minute, but providers commonly enforce it over a shorter rolling window so that a single burst cannot consume the whole minute at once. Sending a minute's worth of traffic in five seconds can return 429 even though the per-minute total is within limits. Spreading requests evenly is more reliable than pacing to the headline figure.
Work out the binding limit above, then set concurrency so the steady-state rate sits somewhere around 70–80% of it. The headroom absorbs retries and variation in request size. Running at exactly the limit guarantees 429s the moment anything varies.
Related: Token Counter · LLM API Cost Calculator