RAG Retrieval Math: Embedding Memory, Vector Index Size, and the Latency Budget Nobody Calculates

Every "we'll just add RAG" decision is a series of arithmetic problems in a trench coat: how much VRAM the embedding model needs, how many bytes the vector index eats, how many distance comparisons a query costs, and whether the whole pipeline lands inside your latency budget. This guide does the math with numbers from primary sources — Hugging Face model cards, NVIDIA datasheets, Qdrant's official documentation, and live API pricing — so you can size the infrastructure before you believe the demo.

The embedding model is the cheap part — until it isn't

Three open embedding models spanning small, mid and large, parameter counts read straight from the Hugging Face repositories (fp16 = 2 bytes per parameter, the same convention as our quantization guide):

The bge-m3 model file is 2,271,145,830 bytes⁴. Dividing its size by four gives roughly 567.8M fp32 values, but serialized file size includes metadata and is not an exact parameter count. The e5-mistral checkpoint ships fp16 safetensors totaling about 14.2 GB³.

The GPU-class consequences at fp16:

• bge-base (0.22 GB) runs on anything — a laptop GPU, a CPU, a Jetson. Memory is not the constraint; throughput is.
• bge-m3 (1.14 GB) fits on any data-center GPU with room to spare; even 8 GB cards carry the model plus large batches.
• e5-mistral-7b (14.22 GB) needs memory for weights, activations and batches. A 24 GB RTX 4090 or A30, a 48 GB L40S, or an A100 40/80 GB provides more room than a 16 GB card; check the actual sequence length and batch.

The expensive tier is not about quality-per-dollar at serving time — it is about the vector width it produces. A 4096-dim model quadruples every downstream storage bill versus 1024-dim bge-m3, at bench scores that improve by single-digit points⁵.

Bytes per vector: where the real storage bill lives

For model-specific context on pooling and vector dimensions, see the EmbeddingGemma 2 architecture guide. Nemotron 3.5 Embed shows how a frozen text tower can preserve an existing text index while a trained visual path enters the same space; compare both single-vector designs with the pplx-embed-v2 guide to asymmetric retrieval and MaxSim. The OlmoEarth v1.3 guide shows a different production case: a trained 128/64-dimensional prefix whose Zarr chunk geometry determines whether truncation also saves network I/O.

Raw embedding storage is embarrassingly simple arithmetic: bytes per vector = dimensionality × bytes per component. As of 2026, Qdrant stores vectors internally as 32-bit floats by default⁶.

Corpus sizing — assume 512-token chunks, so 500M tokens of source text ≈ 1M chunks, 50B tokens ≈ 98M chunks. The index-memory table for bge-class 1024-dim vectors:

And for 4096-dim (7B-class embedder) the same table reaches 1.64 TB raw at 100M chunks — 4× the vector payload. The HNSW graph adds memory: with 4-byte neighbor IDs and up to 2 × m links on layer 0, M=16 gives a rough 128 B per vector for those links alone, or 12.8 GB at 100M vectors. Upper levels, allocator overhead and payload storage add more; this is a sizing example, not an exact Qdrant footprint⁷.

For vector payloads, float32 → uint8 scalar quantization is 4× compression; binary encoding can reach 32× and product quantization depends on its configuration⁶. These ratios exclude index overhead and any original vectors retained for rescoring. Weaviate documents PQ for reducing vector memory⁸, and Milvus offers multiple quantized index types⁹. Recall and latency effects depend on the dataset, query distribution, search settings and rescoring; benchmark them on your corpus.

Latency decomposition: flat brute force vs HNSW

Flat search compares a query with every vector, so its comparison count is N. HNSW is approximate and aims to visit a much smaller portion of the graph, but ef × log₂(N) is not a reliable formula for the number of distance calculations or latency. Its cost depends on graph structure, dimensionality, search parameters and required recall; worst-case instances can take linear work¹⁰.

For 1M 1024-dimensional float32 vectors, an exact scan performs roughly 1M × 2 × 1024 ≈ 2 GFLOP and reads about 4.1 GB of vector payload. Even at an assumed 100 GB/s sustained memory bandwidth, reading that payload alone takes at least ~41 ms; CPU SIMD does not erase memory traffic. Qdrant's full_scan_threshold_kb selects brute force for small candidate sets⁷. Whether exact search remains viable at 1M or 10M vectors depends on hardware, filters, parallelism, QPS and latency targets.

For absolute QPS claims, the only numbers we cite are vendor-run and labeled as such: Qdrant publishes a dedicated HNSW search benchmark repository for exactly this purpose¹¹, and their large-scale tutorial — 400M CLIP vectors, binary quantization, M=6 — reports a specific, reproducible resource breakdown (23.8 GB quantized vectors + 17.9 GB HNSW graph in RAM for the whole 400M)¹². Treat any single-digit-milliseconds-per-query figure from an unconstrained benchmark with skepticism unless the hardware, ef and recall are stated.

Embedding is prefill, not decode — and that changes the math

The KV-cache intuition from our KV cache guide misleads people here, so state it precisely: an embedding forward pass over a 512-token chunk is pure prefill. One shot, all positions at once, no decode steps, no KV cache stored, no memory-bound token-by-token loop. Embedding throughput is compute-bound in a way generation never is.

The FLOPs per chunk are the standard forward-pass estimate, 2 × params × tokens:

• bge-m3 (568M params), 512-token chunk: 2 × 567,755,777 × 512 ≈ 581 GFLOP
• e5-mistral-7b (7.11B params), 512-token chunk: 2 × 7,110,660,096 × 512 ≈ 7,281 GFLOP

Ceilings at spec TFLOPS from NVIDIA datasheets — dense fp16, assuming an honest 50% of peak (model-kernel MFU on well-batched encoder input):

The L40S number deserves its skepticism: 362 TFLOPS is the dense fp16 figure; 1,466 TFLOPS is the sparse FP8 figure¹³. Under the assumed 50% utilization and the linear-layer approximation, a 1M-chunk (500M-token) corpus takes a ~52-minute compute floor (500 × 10⁶ / 159,400 ≈ 52 min). Attention, tokenization, I/O and inefficient batching increase actual time.

Rerankers: the second stage that quietly owns your budget

Hybrid sparse+dense retrieval is the 2026 default because it works — dense vectors miss exact keywords; BM25-style sparse misses paraphrase; bge-m3 returns both from one forward pass². But the accuracy lever people actually feel is the cross-encoder reranker, and it is the most mis-budgeted component in RAG.

A cross-encoder scores each pair separately: full forward pass over concatenated query + candidate. Per candidate pair with a 280M-class reranker (bge-reranker-base, 278,044,931 params¹⁵) over 32 query tokens + 512 doc tokens:

2 × 278M × 544 ≈ 302 GFLOP per pair

Scoring the top-100 candidates costs 100 × 302 GFLOP ≈ 30.3 TFLOP under this linear-layer approximation. On one L40S at an assumed 50% of 362 TFLOPS, that is a ~167 ms compute lower bound, before overhead and attention. Actual GPU and CPU times require measurement; choose the candidate count to meet your measured latency and quality targets.

The two-stage latency budget, per query, at realistic scale:

• Query embedding: measure batching, tokenizer and model latency
• Vector retrieval: measure index, filter selectivity, search width and recall together
• Rerank 100 candidates @ 278M parameters: ~167 ms compute floor under the stated L40S utilization assumption
• Generation: measure prefill and decode separately at the intended context length

In this worked example reranking can dominate retrieval latency; a different index, candidate count or model can change that ordering. Measure every stage of the actual pipeline.

API vs self-host: the break-even is not where you think

Live pricing, per 1M input tokens: OpenAI text-embedding-3-small at $0.02, text-embedding-3-large at $0.13¹⁶. Batch API halves both¹⁶. Embeddings bill input only — no output tokens, because there are none.

For an illustrative rented L40S at an assumed $1.20/hour, the earlier idealized bge-m3 rate of ~159,400 tok/s (50% of stated compute throughput) corresponds to 574M tokens per hour before tokenization, I/O, padding or attention overhead. The resulting compute-only cost per 1M tokens would be:

$1.20 / 574 ≈ $0.0021

That hypothetical compute-only figure is below either API price, but it is not a quote or a measured cost. Hourly billing, startup, CPU/host charges, utilization and model-quality differences can reverse the result.

The table applies the assumptions above with fractional GPU-hour billing; a one-hour minimum would make the 10M-token GPU entry $1.20:

Reading the table without wishful thinking:

• A rented GPU may be cheaper when work can be batched into paid GPU time, the measured model throughput approaches the assumed rate, and extra hosting costs are modest. Data-boundary requirements can also decide the deployment independently of price.
• The API can be simpler and cheaper for sporadic work. At current listed rates, 500M tokens on text-embedding-3-small cost $10 standard or $5 through Batch. Compare that with an actual GPU quote and minimum billing increment.
• At 10M tokens/month, the idealized compute demand in the table is about one minute, but an always-on rental would be billed for a month. The available billing granularity matters more than the arithmetic minimum.

And the quality caveat: quality parity is model-dependent. If you self-host bge-base because it's free, and your baseline is 3-large, the retrieval quality drop eats any infra savings. Match models before comparing prices.

Verdict, with pitfalls stated plainly

There is no universal token-volume crossover. The table itself gives a lower hypothetical rented-GPU cost than the API even at 10M tokens when short rentals are available; a monthly standing rental can instead cost much more. Compare actual prices, billing increments, measured throughput and retrieval quality for the same dataset. Data-boundary requirements may rule out an API regardless of cost.

The honest pitfalls:

• "RAG fixes hallucinations" is a category error. Retrieval grounds generation in what's indexed; if the correct chunk isn't retrieved, the model hallucinates confidently anyway. RAG moves the failure mode from "model doesn't know" to "retrieval missed it" — bounded by recall on your corpus, not benchmark MTEB scores.
• Recall loss is paid twice: approximate HNSW (ef too low) drops recall, quantization drops it further, and each loss compounds silently because the failure surface looks identical to "the model made it up."
• Chunking and model choice interact. Test chunk size, overlap, lexical and dense retrieval on a labeled query set before deciding which component matters most.
• The 7B embedder's 4096-dim vectors cost 4× storage and 4× flat-search time forever. Quality is rented; the vector width is bought.

The math here is the easy part. Every number above is a floor, not a forecast — real pipelines lose to network hops, serialization, and idle utilization. Compute the floors first, then measure the reality against them.

Footnotes