RAG Retrieval Math: Embedding Memory, Vector Index Size, and the Latency Budget Nobody Calculates

Every "we'll just add RAG" decision is a series of arithmetic problems in a trench coat: how much VRAM the embedding model needs, how many bytes the vector index eats, how many distance comparisons a query costs, and whether the whole pipeline lands inside your latency budget. This guide does the math with numbers from primary sources — Hugging Face model cards, NVIDIA datasheets, Qdrant's official documentation, and live API pricing — so you can size the infrastructure before you believe the demo.

The embedding model is the cheap part — until it isn't

Three open embedding models spanning small, mid and large, parameter counts read straight from the Hugging Face repositories (fp16 = 2 bytes per parameter, the same convention as our quantization guide):

The bge-m3 parameter count is verifiable without trusting any blog post: its pytorch_model.bin is 2,271,145,830 bytes in fp32 — divide by 4, get 567.8M parameters⁴. The e5-mistral checkpoint ships fp16 safetensors totaling 14.2 GB, so the weights are the default deployment size³.

The GPU-class consequences at fp16:

• bge-base (0.22 GB) runs on anything — a laptop GPU, a CPU, a Jetson. Memory is not the constraint; throughput is.
• bge-m3 (1.14 GB) fits on any data-center GPU with room to spare; even 8 GB cards carry the model plus large batches.
• e5-mistral-7b (14.22 GB) wants a 24 GB card (L40S, RTX 4090, A30) or an A100 40/80 GB to leave space for activations. A 16 GB card technically fits the weights but leaves almost nothing for batch attention over 512-token inputs.

The expensive tier is not about quality-per-dollar at serving time — it is about the vector width it produces. A 4096-dim model quadruples every downstream storage bill versus 1024-dim bge-m3, at bench scores that improve by single-digit points⁵.

Bytes per vector: where the real storage bill lives

Raw embedding storage is embarrassingly simple arithmetic: bytes per vector = dimensionality × bytes per component. As of 2026, Qdrant stores vectors internally as 32-bit floats by default⁶.

Corpus sizing — assume 512-token chunks, so 500M tokens of source text ≈ 1M chunks, 50B tokens ≈ 98M chunks. The index-memory table for bge-class 1024-dim vectors:

And for 4096-dim (7B-class embedder) the same table reaches 1.64 TB raw at 100M chunks — 4× everything. On top of the vectors, the HNSW graph itself costs RAM: Qdrant documents the exact formula — each node stores roughly m links on upper levels and 2 × m on level 0, each a 4-byte integer, so M=16 costs 2 × 16 × 4 = 128 B per vector, 12.8 GB at 100M vectors⁷.

Compression ratios from the vendors' own docs, not vendor marketing recaps: Qdrant's scalar quantization is float32 → uint8, a hard 4× with "error usually less than 1%"; binary quantization reaches up to 32×; product quantization up to 64× when memory is the top priority⁶. Weaviate documents PQ for reducing memory at scale⁸ and Milvus supports scalar, product and binary/RaBitQ quantization for the same purpose⁹. The honest reading: 4× is nearly free, 32× costs real recall unless you oversample and rescore, and ratios above that are panic mode.

Latency decomposition: flat brute force vs HNSW

Flat search is exact and O(n). HNSW is approximate and roughly logarithmic in theory — but what logarithmic means in comparisons is worth writing out.

Distance comparisons per query, ef = 100 as a realistic search width:

That is the headline: a 100× corpus grows flat search cost 100×, HNSW cost by ~35%. A flat search over 1M 1024-dim float32 vectors is just 1M × 2 × 1024 ≈ 2 GFLOP plus the memory traffic to read 4 GB — millisecond-scale on a modern CPU with SIMD, which is exactly why Qdrant's own docs define a full_scan_threshold_kb below which brute force beats the graph⁷. Flat stops being viable somewhere between 1M and 10M in-RAM vectors; past that, you accept approximate recall.

For absolute QPS claims, the only numbers we cite are vendor-run and labeled as such: Qdrant publishes a dedicated HNSW search benchmark repository for exactly this purpose¹⁰, and their large-scale tutorial — 400M CLIP vectors, binary quantization, M=6 — reports a specific, reproducible resource breakdown (23.8 GB quantized vectors + 17.9 GB HNSW graph in RAM for the whole 400M)¹¹. Treat any single-digit-milliseconds-per-query figure from an unconstrained benchmark with skepticism unless the hardware, ef and recall are stated.

Embedding is prefill, not decode — and that changes the math

The KV-cache intuition from our KV cache guide misleads people here, so state it precisely: an embedding forward pass over a 512-token chunk is pure prefill. One shot, all positions at once, no decode steps, no KV cache stored, no memory-bound token-by-token loop. Embedding throughput is compute-bound in a way generation never is.

The FLOPs per chunk are the standard forward-pass estimate, 2 × params × tokens:

• bge-m3 (568M params), 512-token chunk: 2 × 567,755,777 × 512 ≈ 581 GFLOP
• e5-mistral-7b (7.11B params), 512-token chunk: 2 × 7,110,660,096 × 512 ≈ 7,281 GFLOP

Ceilings at spec TFLOPS from NVIDIA datasheets — dense fp16, assuming an honest 50% of peak (model-kernel MFU on well-batched encoder input):

The L40S number deserves its skepticism: 362 TFLOPS is the dense fp16 figure behind the 1,466-TFLOPS sparse headline¹². Even at half utilization, one L40S embeds a 1M-chunk (500M-token) corpus with bge-m3 in under an hour of pure compute (500 × 10⁶ / 159,400 ≈ 52 min). Memory is not the bottleneck; scheduling data movement into the batches is.

Rerankers: the second stage that quietly owns your budget

Hybrid sparse+dense retrieval is the 2026 default because it works — dense vectors miss exact keywords; BM25-style sparse misses paraphrase; bge-m3 returns both from one forward pass². But the accuracy lever people actually feel is the cross-encoder reranker, and it is the most mis-budgeted component in RAG.

A cross-encoder scores each pair separately: full forward pass over concatenated query + candidate. Per candidate pair with a 280M-class reranker (bge-reranker-base, 278,044,931 params¹⁴) over 32 query tokens + 512 doc tokens:

2 × 278M × 544 ≈ 302 GFLOP per pair

Scoring the top-100 candidates from retrieval costs 100 × 302 GFLOP ≈ 30.3 TFLOP — on one L40S at 50% MFU that is ~167 ms per query, before the generation call has started. The same 100-pair workload on a CPU-only path is minutes. This is why production pipelines rerank 20–50 candidates, not 100, and why reranker choice is a latency decision, not just a quality one.

The two-stage latency budget, per query, at realistic scale:

• Query embedding (bge-m3 over 32 tokens): sub-millisecond-class on GPU, ~ms-class on CPU
• HNSW retrieval over 10M vectors: single-digit ms (recall < 1.0; you chose that)
• Rerank 100 candidates @ 280M params: ~100–170 ms GPU
• Generation over 8k context: seconds — and as the KV cache guide shows, dozens of GB if mismanaged

The uncomfortable observation: the reranker, not the vector search, dominates retrieval latency, and the generator dominates the whole pipeline. Vector search at 10M scale is not your problem. Everything around it is.

API vs self-host: the break-even is not where you think

Live pricing, per 1M input tokens: OpenAI text-embedding-3-small at $0.02, text-embedding-3-large at $0.13¹⁵. Batch API halves both¹⁵. Embeddings bill input only — no output tokens, because there are none.

Self-hosting on one L40S at a typical $1.20/hour on-demand cloud rate: bge-m3 at ~159,400 tok/s (50% MFU) embeds 574M tokens per hour. Cost per 1M tokens:

$1.20 / 574 ≈ $0.0021

That is ~10× cheaper than text-embedding-3-small and ~60× cheaper than 3-large — per token, while the GPU is running at utilization. The catch: a GPU billed by the hour that embeds your corpus in an afternoon and then sits idle is not cheaper than anything.

So the honest break-even framing is utilization, not corpus size:

Reading the table without wishful thinking:

• Self-hosting wins when your pipeline re-embeds continuously (crawls, churn, re-indexing on model swaps), when data can't leave the boundary (our EU-AI-Act-adjacent guides cover why European deployments care), or when the volume makes a dedicated card run hot: at multi-billion tokens per month re-embedded, one L40S pays for itself monthly against even the cheapest API.
• The API wins at one-shot indexing: embedding 500M tokens once costs $5 with 3-small batch (half the $10 standard price) — a twentieth of one month of even a cheap GPU lease. Nobody should buy a GPU for a single corpus pass.
• Continuous corollary: at 10M tok/month, the "GPU cost" is 1 minute of compute. Rent it by the second (serverless GPU) or use the API; a standing instance is the only way to lose this comparison.

And the quality caveat: quality parity is model-dependent. If you self-host bge-base because it's free, and your baseline is 3-large, the retrieval quality drop eats any infra savings. Match models before comparing prices.

Verdict, with pitfalls stated plainly

Self-hosted RAG infrastructure math beats an API under three conditions: sustained embedding volume (≥ single-digit billion tokens/month re-processed), hard data-boundary requirements, or the 7B-class embedder quality tier where no API offers an equivalent-quality model you're allowed to post-process. Below one billion tokens per month of embedding work, and without a data boundary, the API is arithmetically unbeatable — $2–20/month against any GPU contract.

The honest pitfalls:

• "RAG fixes hallucinations" is a category error. Retrieval grounds generation in what's indexed; if the correct chunk isn't retrieved, the model hallucinates confidently anyway. RAG moves the failure mode from "model doesn't know" to "retrieval missed it" — bounded by recall on your corpus, not benchmark MTEB scores.
• Recall loss is paid twice: approximate HNSW (ef too low) drops recall, quantization drops it further, and each loss compounds silently because the failure surface looks identical to "the model made it up."
• Chunking strategy dominates model choice before either is benchmarked — 512-token chunks with no overlap lose to naive BM25 with better chunking.
• The 7B embedder's 4096-dim vectors cost 4× storage and 4× flat-search time forever. Quality is rented; the vector width is bought.

The math here is the easy part. Every number above is a floor, not a forecast — real pipelines lose to network hops, serialization, and idle utilization. Compute the floors first, then measure the reality against them.

Footnotes