What a Small RAG Pipeline Costs Per Month

By J. Wren · Published 2026-09-15

Most RAG (retrieval-augmented generation) cost estimates skip straight to "it depends," which is true and not very useful. Here is one fully specified pipeline, priced end to end with actual current API rates: what it costs to embed the initial document set, what it costs to embed queries every month, and what it costs to generate answers, compared across three chat models. Every number below comes out of a script that read this site's pricing data as of 2026-09-14 — change the assumptions or re-run it after prices move, and the numbers change with it.

The pipeline, fully specified

Change any of these and the pipeline is a different one — that is the point of writing them all down instead of quoting a single number that only applies to this exact case.

Step 1: how many chunks

800 words per document at 0.75 words/token is about 1,067 tokens per document. With a 500-token chunk size and 50-token overlap (a 450-token step between chunk starts), that comes out to 3 chunks per document, or 1,500 chunks across all 500 documents.

Step 2: initial embedding cost (one-time)

Embedding all 1,500 chunks at 500 tokens each is 750,000 tokens, run once. At current rates:

At this document count, the one-time embedding cost is small enough to round to a few cents regardless of which model is used. It stops being negligible only at a much larger corpus, or if documents are re-embedded frequently.

Step 3: monthly query embedding cost

Every incoming query also needs to be embedded before it can be matched against the stored chunks: 20,000 queries × 27 tokens = 540,000 tokens a month.

Step 4: generation cost — the part that actually matters

Each query's prompt to the chat model carries the 150-token system prompt, the 27-token question, and the 5 retrieved chunks at 500 tokens each: 150 + 27 + 2,500 = 2,677 input tokens per query. Across 20,000 queries a month that is 53,540,000 input tokens and, at 300 output tokens per answer, 6,000,000 output tokens.

Generation dominates the bill by a wide margin — the retrieved context (2,500 of the 2,677 input tokens per query, 93%) is doing almost all of the work, and almost all of the cost. The embedding steps above, one-time and monthly combined, come to under twenty cents regardless of model choice; the choice of chat model swings the monthly total by more than $100.

Putting it together: total monthly recurring cost

Combining each chat model's generation cost with the cheapest query-embedding option (OpenAI text-embedding-3-small, $0.01/month at this volume):

The one-time initial embedding cost (1–11 cents depending on model, see Step 2) is separate from this recurring figure and only happens once per document set.

Where this estimate would move in practice

Three levers matter more than any of the specific dollar figures above. Chunks retrieved per query scales the input cost directly — dropping from 5 to 3 chunks cuts roughly 40% off the 2,500-token retrieval block, which is most of the prompt. Prompt caching is not modeled here at all; if the system prompt and any shared context is cached, the effective input rate on that portion drops well below list price. Answer length matters less than it looks like it should here, because output tokens are only 6,000,000 against 53,540,000 input tokens — this particular pipeline is retrieval-heavy, not generation-heavy, which is common for RAG and not universal. In dollar terms it is still not trivial, though: output accounts for 36% of the Gemini and Claude Sonnet totals above and 40% of GPT-5.6 Terra's, because output tokens bill at several times the input rate even though there are far fewer of them.

How to reproduce or adapt this

The full calculation is in rag_cost_example.py, published alongside its results.json, reading directly from the same pricing.json that powers the calculators on this site. Every constant at the top of the script — document count, chunk size, query volume, retrieved chunks, answer length — is a variable you can change to model a different pipeline; the rest of the script recomputes from there.

python rag_cost_example.py

For a one-off estimate without running a script, the embedding cost calculator handles the embedding side, and the LLM API cost calculator handles generation once you know your own token counts. If token counts are the uncertain part, start with the token counter, and check whether the retrieved context fits the model's window with the context window calculator. Current per-vendor rates: Claude, GPT, Gemini. This pipeline's rates are as of 2026-09-14; check the pricing change log for anything since, and confirm current numbers there before budgeting.

FAQ

Does this include the vector database? No. Hosting or a managed vector database is a separate infrastructure cost this article does not estimate.

Why is Gemini 3.6 Flash so much cheaper here? It has the lowest input and output rate of the three chat models compared, and this pipeline is dominated by input tokens (93% of every prompt is retrieved context) — a model's input rate matters more than its output rate in a retrieval-heavy pipeline like this one.

What if my documents are longer or shorter than 800 words? Re-run the script with a different WORDS_PER_DOC value; chunk count and every downstream cost scale from that number.