Gemini vs GPT Token Counts

By J. Wren · Published 2026-10-13

The earlier article on this site, Does 4 Characters Really Equal 1 Token?, counted tokens with two OpenAI tokenizers and nothing else, and it said so: whatever Gemini does was left open. This article takes the Gemini half of that gap. I kept the same fifty text samples, asked Google's countTokens endpoint how many tokens each of the four Gemini model ids on this site's pricing pages would count, and put the result next to the OpenAI counts and then next to the prices.

Three things came out of it. On English prose the two families land within about one percent of each other: Gemini used 1.009 tokens for every o200k_base token. On Chinese, Gemini used fewer tokens than o200k_base — 0.762 of them on Traditional Chinese and 0.845 on Simplified. On Python code and JSON it used more: 1.226 and 1.160. All four Gemini ids returned exactly the same count on all fifty samples, so they share one column below, but each was called and recorded on its own.

How the counts were made

The fifty samples are the ones from the earlier article: ten segments each of English prose, Traditional Chinese, Simplified Chinese, Python code and JSON, all written for the measurement rather than copied from anywhere. The new script imports them from the earlier script instead of keeping a second copy, so the two articles cannot drift apart. For every segment it records the character count, the token count from each OpenAI tokenizer, and the count the Gemini API returns for each of four ids: gemini-3.1-pro-preview, gemini-3.6-flash, gemini-3.5-flash-lite and gemini-3.1-flash-lite. The counts were taken on 2026-10-13.

Each id was called separately, and nothing was substituted: had one failed, it would be marked n/a in the results file and left out of this article, and no other model's number would have stood in for it. All four succeeded. I ran the whole measurement more than once and every count came back identical. The endpoint returns only a token total — no model id comes back with the response — so the id I requested is the only id recorded.

One detail matters for short strings. The endpoint adds a fixed overhead to each request: an empty string comes back as 1 token on all four ids. Segments this short would have their Gemini totals inflated by a few percent if that were left in, so the script measures the empty-string baseline for every id and subtracts it. The raw counts stay in the results file under total_tokens_raw_with_overhead. Every Gemini figure below is the net count.

Table 1 averages the per-sample characters-per-token ratios inside each category, exactly as the earlier article did, which is why the English o200k_base figure matches the one there. Tables 2 and 3 use pooled totals — all characters divided by all tokens — because a bill depends on totals, not on an average of ratios.

Results: characters per token

Figure 1 plots Table 1. The four Gemini ids are drawn as one bar because their per-sample counts were identical in every category; the table lists them separately.

Figure 1 — average characters per token, by text type and tokenizer

Table 1 — average characters per token (mean of the ten per-sample ratios in each category).

Table 2 — total tokens for the ten segments in each category (Gemini = net of the 1-token overhead; the same for all four ids).

What the numbers say

English prose is close to a tie. Over the ten segments Gemini counted 325 tokens, o200k_base 322 and cl100k_base 327. In characters per token that is 5.185 for Gemini against 5.233 and 5.153 for the two OpenAI encodings — all three above the 4 of the usual rule of thumb. For a plain-English prompt the choice of family barely moves the token count.

Chinese is where they separate. The Traditional Chinese segments total 482 characters; Gemini counted 311 tokens for them and o200k_base 408. The Simplified Chinese segments total 479 characters, counted as 289 by Gemini and 342 by o200k_base. As characters per token, Gemini is at 1.548 (Traditional) and 1.660 (Simplified), against 1.178 and 1.401 for o200k_base. Both are far from the 4 of the rule of thumb, but the same Chinese text costs Gemini noticeably fewer tokens than it costs the OpenAI encoding.

Code and JSON go the other way. Python code totals 645 tokens on Gemini against 526 on o200k_base (3.075 against 3.742 characters per token). JSON is 391 against 337 (2.284 against 2.647). Structured text full of punctuation and short identifiers is where the OpenAI encoding was more compact on these samples. A prompt that is mostly JSON payloads or source files was counted as 1.160 to 1.226 times as long by Gemini as by o200k_base on these samples.

These are counts, not prices. A token count only becomes a cost once it meets a rate, and the rates differ by model, so the next section does that step.

From tokens to money

For input text, the cost of one million characters is the model's input rate divided by its pooled characters per token. A GPT row uses o200k_base (the assumption described above). A Gemini row uses that model's own measured ratio and nothing borrowed from another id. The rates are the ones this site's data held on 2026-10-13; they move, so check the change log or the calculators before relying on a dollar figure from this table. Claude rows are left out because no Claude count exists here.

Table 3 — input cost in USD per one million characters of each text type.

Two pairs of models share a listed input rate, which isolates the tokenizer: Gemini 3.1 Pro and GPT-5.6 Terra at $2.00 per million input tokens, and Gemini 3.6 Flash and GPT-5.4 mini at $0.75. Inside each pair, Gemini's cost per character is 1.009 times the GPT model's on English prose, 0.762 on Traditional Chinese, 0.845 on Simplified Chinese, 1.226 on Python code and 1.160 on JSON. These are the same figures as the last column of Table 2, as they must be when the rate is equal. Same sticker price, and which one is cheaper depends on what you send.

The ranking can also flip across families when the sticker prices differ. GPT-5.4 nano, at a lower input rate, is cheaper than Gemini 3.1 Flash-Lite on English prose — $0.0383 against $0.0483 per million characters — and more expensive on Traditional Chinese, $0.1693 against $0.1613. On Simplified Chinese nano stays ahead, $0.1428 against $0.1508, so the flip is specific to the text, not to the language group as a whole.

What this does not show

Ten short segments per category is enough to show that the families differ and in which direction; it is not enough to pin down a multiplier for your workload. Longer documents, other registers of Chinese, minified code and deeply nested JSON could all land elsewhere. The samples are my own writing, so nothing here says how either tokenizer treats a specific published text.

That the four Gemini ids agreed on every sample tells me only that they agreed on these fifty samples. It does not establish that they share a tokenizer, and a future model id could count differently, which is why the script calls each id rather than reusing one result. The same caution applies to the preview id in the list. Google's Gemma 3 technical report says Gemma 3 uses the Gemini 2.0 tokenizer; I found no Google statement covering the 3.x ids listed here.

The endpoint counts the text I sent. A real request adds system instructions, tool definitions, conversation history and sometimes images, none of which is in these numbers, and I did not compare the endpoint's totals against the usage figures from a billed request. Only the input side is covered: output tokens, and any internal reasoning tokens a model bills as output, are outside this measurement. And the GPT-5 models are assumed to use o200k_base, as noted at the top.

How to reproduce this

The script and the full per-sample results are published next to the earlier ones: measure_gemini_vs_gpt_tokens.py and measure_gemini_vs_gpt_tokens.results.json. The script imports the samples from measure_chars_per_token.py, so download both files into one folder.

pip install tiktoken google-genai
export GEMINI_API_KEY=your-own-key
python measure_gemini_vs_gpt_tokens.py

You need your own Gemini API key; the script reads it from the environment and never writes it anywhere. It prints a summary and rewrites the results file. The OpenAI counts are deterministic and offline. For the Gemini counts, a re-run should reproduce the totals in Table 2 unless Google has changed a model's tokenizer in the meantime — if it does not, that is itself worth knowing.

To turn a token count into dollars, use the LLM API cost calculator or the per-vendor pages for Gemini and GPT. For a quick count of a specific prompt with the OpenAI encodings, the token counter runs them in your browser. For how the daily price data behind Table 3 is kept current, see the pricing change log.