Understanding the Real Costs of Running a RAG System in Production

Almost every team embarking on a Retrieval-Augmented Generation (RAG) project tends to worry about the wrong cost elements. Many assume that embedding 100,000 documents is costly, that storing hundreds of thousands of vectors requires specialized infrastructure, and that the ingestion pipeline consumes the budget. These assumptions, however, are incorrect, as demonstrated by the calculations below.

Cost Breakdown for a Production RAG System

A typical production RAG system working with a 100,000-document corpus incurs the following costs on a cloud platform, broken down item by item: embedding the entire corpus has a one-time cost of $6.86. This is calculated from 100,000 documents averaging 1,500 tokens each, resulting in 150M tokens of text, which becomes 171.4M tokens after chunking overlap. This is billed at a published rate of $0.04 per million tokens (171.4M × $0.04/1M = $6.86). Storing vectors adds about a dollar a month to a managed database. The persistent cost, month after month, is the reading of tokens at answer time. Reranking and answer generation account for over 99% of the model expenses at every traffic level, and these are the only costs that increase with traffic (the database cost is a fixed line item, dominant at low traffic and accounts for 6% of the bill at high traffic). By carefully selecting the number of chunks read per query and the model used, a RAG system handling 100,000 documents can operate at approximately $100 per month with 1,000 queries per day.

This article provides a detailed cost model, supported by the working pipeline upon which it is based: chunking, embeddings, vector storage in a managed database with vector extensions, retrieval, reranking, and serving through a serverless inference endpoint. Every number has its formula presented so you can adjust the model to your corpus size, token counts, and traffic.

Important Considerations

This is a cost model, not a report from a production deployment. The prices are current (as verified in August 2026), but the traffic-level totals are calculated based on stated assumptions, not actual bills. Your costs will vary with different token counts, and the formulas provided make it easy to verify. Always check the latest prices before making financial commitments.

Key Assumptions

• Corpus: 100,000 documents, averaging 1,500 tokens each, chunked at 512 tokens with a 64-token overlap into approximately 335,000 chunks.
• Per Query: 32-token question embedding, reranking of 20 retrieved candidates, and answer generation over the top 5 chunks.
• Models: Various embedding and generation models with specific costs per million tokens.

Ingestion and Monthly Costs

One-time Ingestion Costs

• Embeddings (Qwen3 Embedding 0.6B): 171.4M tokens × $0.04/1M = $6.86
• Optional Chunk Enrichment: Varies from $9.46 to $18.92 depending on batch inference discounts.

Ingesting 100,000 documents costs under $10 without enrichment and less than $20 with it. If the corpus size doubles, these costs double accordingly.

Monthly Costs at Different Traffic Levels

| Item | 1K queries/day | 10K queries/day | 100K queries/day | |----------------------------|----------------|-----------------|------------------| | Query embeddings | $0.04 | $0.38 | $3.84 | | Reranking (DeepSeek V4 Flash)| $21.50 | $215 | $2,150 | | Answer generation | $16.05 | $161 | $1,605 | | Managed Database | $60 | $120 | $240 | | Total per month | ~$98 | ~$496 | ~$4,000 |

Using alternative models like Llama 3.3 70B for generation can significantly increase costs.

Key Cost Influencers

1. Embeddings and Storage: These are relatively inexpensive components of the system.
2. Model Selection for Generation: Choosing the right model can result in a fourfold cost difference.
3. Reranker Costs: At high traffic, reranking becomes a significant cost due to the number of tokens processed.

Strategies to Lower Costs

• Reranking: Reduce the number of candidates or use a cheaper managed reranker.
• Answer Generation: Implement prompt caching and control token usage.

Conclusion

The perception of RAG systems as expensive is largely due to misunderstandings about where costs are incurred. The most significant costs arise from reranking and answer generation, which scale with traffic. By focusing on selecting efficient models and managing traffic-related costs, the overall expense can be effectively controlled.