Does 4 Characters Really Equal 1 Token?

By J. Wren · Published 2026-09-15 · Updated 2026-10-13

"1 token ≈ 4 characters" is the rule of thumb repeated on this site's own cost calculator and on nearly every other one like it. It is a genuinely useful shortcut for a rough estimate, and it is also, as stated, only true for one kind of text. I ran the actual numbers through two real OpenAI tokenizers across five categories of text — English prose, Traditional Chinese, Simplified Chinese, Python code, and JSON — ten samples each, fifty samples total, and counted characters and tokens directly. Here is what came out.

Method

All fifty samples were written for this measurement — original text, not copied from any book, article or codebase — to avoid any question of what a copyrighted source's actual token count reveals. Each category has ten segments of ordinary length (roughly one to three sentences of prose, or a short function, or one JSON object). For every segment I recorded the character count (Python's len() on the string) and the token count from each tokenizer, then computed characters ÷ tokens per segment. The table below reports the average, minimum and maximum of that ratio across the ten segments in each category.

Results: characters per token, by category

What this means

English prose is close to the rule of thumb, but a little above it. Both tokenizers measured around 5.1–5.2 characters per token on my samples, not 4. "4 characters per token" is conservative for plain English sentences — it will slightly overestimate token counts, which for a cost calculator means slightly overestimating the bill, the safer direction to be wrong in.

Chinese breaks the rule badly, in the opposite direction. Traditional Chinese measured well under 1 character per token with cl100k_base (0.78), and Simplified Chinese came in at about 1 (0.99) — meaning each Chinese character typically costs at least one full token, not a quarter of one. Applying "4 characters ≈ 1 token" to Chinese text would understate the token count by roughly 4 to 5 times on these samples with cl100k_base. o200k_base is noticeably better at Chinese (1.18–1.40 characters/token, still roughly a 3x understatement) but still nowhere near 4. If a workload processes Chinese input at volume, a character-based estimate is not a safe substitute for an actual tokenizer count.

Code and JSON sit in between. Python code measured 3.7–3.8 characters per token — close enough to 4 that the rule of thumb is only a mild underestimate here, the opposite direction from English's mild overestimate. JSON measured 2.6–2.7, meaningfully below 4: punctuation-heavy, quote-and-brace structure tends to tokenize less efficiently than prose. A prompt built mostly of JSON payloads will cost more tokens per character than the same byte count of English explanation.

The two tokenizers barely differ on English, code and JSON — within a few percent of each other on every category except Chinese, where o200k_base's larger vocabulary gives it a real advantage. That gap is the one place in this dataset where which specific OpenAI model you use actually changes the answer by a meaningful margin.

What I am not claiming

Ten samples per category is enough to show that the 4-characters rule behaves differently across text types, not enough to pin down a precise multiplier for any one of them — a larger, more varied corpus would likely narrow or shift these ranges somewhat. I also only tested prose-register Chinese and idiomatic Python; dense technical Chinese, minified code or deeply nested JSON could tokenize differently again. And to repeat the point from the top: no Claude or Gemini numbers appear anywhere in this article, measured or estimated.

How to reproduce this

The measurement script and full per-sample results are published at /data/methods/measure_chars_per_token.py and /data/methods/measure_chars_per_token.results.json:

pip install tiktoken
python measure_chars_per_token.py

The script defines all fifty sample segments inline, runs both tokenizers over each one, and writes a JSON file with the per-sample character count, token count and ratio, plus the aggregated averages shown above. Swap in your own text to get a ratio for your actual workload instead of mine — that is a better number than any rule of thumb, including this article's.

For a quick estimate on a specific prompt rather than a category average, use the token counter — it also runs on the chars/4 heuristic rather than a live tokenizer, so treat it as a fast approximation, not an exact count. To turn a token count into a dollar figure, the LLM API cost calculator, or the per-vendor pages for Claude, GPT and Gemini, apply current rates to whatever count you land on. If the workload involves retrieval, the embedding cost calculator and context window calculator are the next stop.

Follow-up: Gemini

This article does not cover Gemini. The same fifty samples were counted on four Gemini model ids and compared with the OpenAI counts above, with a cost-per-character table, in Gemini vs GPT Token Counts.

FAQ

Does this apply to Claude or Gemini token counts? Not by this article. Claude is not measured on this site; Gemini was counted separately on the same fifty samples in Gemini vs GPT Token Counts. Both use different tokenizers than the two measured here, so the figures above describe OpenAI's encodings only.

Which tokenizer should I use for GPT-5.6 or GPT-5.4? Check the model provider's current documentation for which encoding a given model family uses; this article measured cl100k_base and o200k_base specifically, not a full mapping of every current model to its encoding.

Is 4 characters per token ever exactly right? Not in any category I measured — Python code came closest, at roughly 3.7–3.8, not 4.