Check if your prompt fits a model's context window — and how much room is left for the response.
A model's context window is the maximum number of tokens it can consider at once — your prompt plus the response it generates. If your input plus the reserved output exceeds the window, the request fails or the model truncates earlier content. Always leave headroom for the answer.
| Part | Usually forgotten because |
|---|---|
| System prompt | It is set once in code and never seen again, but it is resent with every request |
| Conversation history | It grows every turn — the same chat costs more tokens the longer it runs |
| Tool and function definitions | Schemas are sent as part of the prompt; a dozen tools can run to thousands of tokens |
| Retrieved documents | Usually the largest single block, and the easiest to over-fetch |
| The response | Generated inside the same budget, not on top of it |
A prompt that technically fits can still produce worse answers than a shorter one. Models attend unevenly across a long context, and material buried in the middle of a very large prompt tends to be used less reliably than material near the beginning or the end. Filling a million-token window because it is available is rarely the best move.
Cost and latency scale with what you send, too. Input tokens are billed on every request, so a system prompt that is twice as long as it needs to be is a permanent tax on every call — see the LLM API Cost Calculator for what that adds up to.
Trim before you chunk. Removing boilerplate, duplicated context and stale conversation turns is cheaper and less lossy than splitting the job.
Retrieve less, more precisely. Sending the five most relevant passages usually beats sending fifty, both for cost and for answer quality.
Summarise older turns. In long conversations, replace early history with a compact summary and keep recent turns verbatim.
Split the task. Map over the pieces and reduce the results, rather than trying to force one giant request.
Roughly one token per four English characters, or about three quarters of a word — but the ratio shifts with code, non-English text and unusual formatting, all of which tokenise less efficiently. Use the Token Counter for an estimate rather than counting words.
Behaviour depends on the API. Most return an error rather than guessing what to drop, which is the safer outcome because it is visible. Some client libraries and chat frameworks silently trim the oldest messages instead, which looks like the model forgetting things partway through a conversation.
No. Caching reduces what you pay for repeated context, and often the latency, but cached tokens still occupy the context window. It is a cost optimisation, not extra room.