llama.cpp / Ollama vs vLLM: Which One, When, and Why

Local LLM inference has converged on three stacks that are constantly confused with each other: llama.cpp, Ollama, and vLLM. They are not competitors at the same layer — two of them literally wrap or complement each other. This guide pins down what each one is, where the concurrency and memory models diverge, and which to pick per scenario.

What each layer actually is

llama.cpp is the C/C++ inference engine: a GGUF-native runtime with backends for CUDA, Metal, Vulkan, ROCm, and plain CPU, plus llama-server, a built-in HTTP server with OpenAI- and Anthropic-compatible routes, continuous batching, and parallel decoding.¹

Ollama is a model distribution and runtime layer, not an engine of its own (mostly). Historically it wrapped a pinned llama.cpp build — the LLAMA_CPP_VERSION variable in the Ollama repo pins the exact upstream source, patched via llama/compat/.² On top of that pin Ollama has added its own engines: a multimodal engine that runs vision models (Llama 4, Gemma 3, Qwen 2.5 VL) directly on the GGML tensor library instead of gluing separate text-decoder and vision-encoder processes together,³ and an MLX-based engine for Apple Silicon that Ollama put into preview in March 2026, replacing the llama.cpp Metal path for accelerated models.⁴

vLLM is a Python/CUDA serving engine built around PagedAttention: KV-cache memory managed OS-style in fixed-size blocks with logical-to-physical block tables, giving near-zero cache waste and 2–4× throughput over the systems it was benchmarked against in the original paper.⁵ Since the V1 re-architecture it enables prefix caching by default (under 1% throughput loss at 0% hit rate) and reaches up to 1.7× higher throughput than V0.⁶

Concurrency models: fixed slots vs a shared pool

This is the single biggest behavioral difference, and the source of the worst gotcha.

llama-server: --parallel N fixed slots, --ctx-size split across them

llama-server -np N creates N slots. Current master defaults: -np/--parallel default is -1 (auto) and -cb/--cont-batching (continuous batching) is enabled by default; -b/--batch-size defaults to 2048 and -ub/--ubatch-size to 512. --ctx-size (default 0 = loaded from model) is the total context, and the KV budget is divided across slots.¹

The gotcha: --ctx-size is the total, not per-slot. With -c 8192 -np 4, every slot gets 8192/4 = 2048 tokens of KV space — your 8k prompt silently truncates or errors. If you actually want 8k per slot across 4 slots, you must pass -c 32768 -np 4. This has tripped up enough users that upstream added an issue-and-fix cycle and eventually a per-slot control (ctx-per-slot / --kv-unified-per-slot on current master) to set the per-slot context limit directly.⁷

vLLM: one paged KV pool, continuous batching

vLLM keeps a single shared KV pool and allocates blocks on demand; requests join and leave the batch at every decode step. Raising concurrency costs nothing when the server is idle because no per-request context is pre-reserved: a request's KV cache grows token-by-token in physical blocks, freed immediately when the request finishes.⁵ --max-num-seqs limits batch size; the true limit on concurrent requests is "how many sequences fit their accumulated KV into the pool."

Ollama: slots like llama.cpp, sized by defaults

Ollama inherits the slot model: OLLAMA_NUM_PARALLEL (default 1) is the number of parallel requests each model processes, and required RAM scales by OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH.⁸ With the default of 1, concurrent requests queue; each additional slot multiplies the KV allocation just like --parallel does in llama-server.

GGUF: native home vs experimental guest

GGUF is llama.cpp's native format — the reference implementation, with the full quant family (Q2_K through Q8_0, i-quants, imatrix calibration).⁹

vLLM's GGUF support is a different story, in its own docs' words: "GGUF support in vLLM is highly experimental and under-optimized at the moment, it might be incompatible with other features." GGUF loading has migrated out of core into the OOT vllm-gguf-plugin package (uv pip install vllm-gguf-plugin), only single-file GGUFs are supported, and the docs recommend passing the base model's tokenizer explicitly, e.g.¹⁰:

vllm serve unsloth/Qwen3-0.6B-GGUF:Q4_K_M --tokenizer Qwen/Qwen3-0.6B

Practical rule: on vLLM, GGUF is a memory-footprint reduction, not a performance path. For high-throughput serving use safetensors with a first-class quantization scheme (AWQ, GPTQ, FP8); use GGUF only when that model exists only as GGUF.

Defaults that bite

Ollama context defaults are VRAM-tiered. Current docs: < 24 GiB VRAM → 4k context; 24–48 GiB → 32k; ≥ 48 GiB → 256k, overridable with OLLAMA_CONTEXT_LENGTH (older releases used a flat 4096 default, and 4096 is still what the FAQ's override examples show).¹¹ If your 24 GB card picks a 32k default you didn't ask for, KV cache grows accordingly — check ollama ps (its CONTEXT column) and set num_ctx per request or the env var globally.

Ollama KV cache type defaults to f16. OLLAMA_KV_CACHE_TYPE accepts f16 (default), q8_0 (~half the KV memory, tiny quality loss), or q4_0 (~quarter, more noticeable loss) — and requires flash attention to be enabled to take effect.⁸

OLLAMA_NUM_PARALLEL defaults to 1 — simultaneous requests queue rather than run concurrently, and raising it multiplies KV memory by N.⁸

vLLM gpu_memory_utilization defaults to 0.9 — vLLM claims up to 90% of the card's total memory at startup (per-instance, not per-process-coordinated), then subtracts weights, activation peak, and CUDA-graph pools; the KV cache is the residual. On a 24 GB card shared with a display or other processes, 0.9 will OOM at init — lower it (0.80–0.85) or pin --max-model-len.¹²

Worked example: 14B Q4_K_M, 4 slots × 8k, on a 24 GB card

Take Qwen2.5-14B (48 layers, GQA with 8 KV heads, hidden size 5120 across 40 heads → head dim 128, 14.7B total parameters).¹³ We compute the budget using the site's KV formula (see the KV cache guide and inference math), plugging in llama.cpp's measured Q4_K_M ≈ 4.85 bits-per-weight for this weight class.⁹

Weights at 4.85 bpw:

KV cache per token (f16, 2 bytes/value), using $2 \times L \times H_{KV} \times d_{head} \times b_{dtype} \times s$:

Per slot at 8192 tokens, and total for 4 slots:

It fits — with room to spare, which is the point: on a 24 GB card a 14B Q4_K_M with 4 concurrent 8k conversations is comfortably inside budget, and the ~8.65 GB headroom — 24 decimal GB on the sticker minus the 14.30 GiB subtotal (15.35 decimal GB) — covers compute scratch (dependent on batch size and backend) plus the CUDA context. Two levers if you need more slots or context: -ctk q8_0 -ctv q8_0 halves KV to 3.0 GiB (llama-server flags, f16 default¹), the same trade Ollama's OLLAMA_KV_CACHE_TYPE=q8_0 makes. For a full sizing workflow see the VRAM calculator explained and quantization guide; the interactive tool is at LLM inference calculator.

And remember the gotcha from above: to actually get 8k per slot you'd start llama-server -m model-Q4_K_M.gguf -c 32768 -np 4, not -c 8192.

Decision table

Rule of thumb: llama.cpp when you need to touch the knobs or fit weird hardware; Ollama when you want zero-friction local use; vLLM when other people or services are hitting the endpoint and throughput per GPU matters.