llama.cpp / Ollama vs vLLM: Which One, When, and Why

Local LLM inference has converged on three stacks that are constantly confused with each other: llama.cpp, Ollama, and vLLM. They are not competitors at the same layer — two of them literally wrap or complement each other. This guide pins down what each one is, where the concurrency and memory models diverge, and which to pick per scenario.

What each layer actually is

llama.cpp is the C/C++ inference engine: a GGUF-native runtime with backends for CUDA, Metal, Vulkan, ROCm, and plain CPU, plus llama-server, a built-in HTTP server with OpenAI- and Anthropic-compatible routes, continuous batching, and parallel decoding.¹

Ollama is a model distribution and runtime layer, not an engine of its own (mostly). Historically it wrapped a pinned llama.cpp build — the LLAMA_CPP_VERSION variable in the Ollama repo pins the exact upstream source, patched via llama/compat/.² On top of that pin Ollama has added its own engines: a multimodal engine that runs vision models (Llama 4, Gemma 3, Qwen 2.5 VL) directly on the GGML tensor library instead of gluing separate text-decoder and vision-encoder processes together,³ and an MLX-based engine for Apple Silicon that Ollama put into preview in March 2026, replacing the llama.cpp Metal path for accelerated models.⁴

vLLM is a Python/CUDA serving engine built around PagedAttention: KV-cache memory managed OS-style in fixed-size blocks with logical-to-physical block tables, giving near-zero cache waste and 2–4× throughput over the systems it was benchmarked against in the original paper.⁵ Since the V1 re-architecture it enables prefix caching by default (under 1% throughput loss at 0% hit rate) and reaches up to 1.7× higher throughput than V0.⁶

Concurrency models: slots and KV allocation

This is the single biggest behavioral difference, and the source of the worst gotcha.

llama-server: --parallel N slots, with optional unified KV

llama-server -np N creates N slots. Current master defaults: -np/--parallel default is -1 (auto) and -cb/--cont-batching (continuous batching) is enabled by default; -b/--batch-size defaults to 2048 and -ub/--ubatch-size to 512. --ctx-size (default 0 = loaded from model) sets the total KV capacity. With --no-kv-unified, it is divided across slots; with --kv-unified, slots share one pool. The current auto-slot default enables unified KV.¹

The gotcha applies to non-unified mode: with -c 8192 -np 4 --no-kv-unified, each slot gets 8192/4 = 2048 tokens of KV capacity. For four simultaneous 8k slots in that mode, use -c 32768 -np 4 --no-kv-unified. With --kv-unified, the 8192-token pool is shared rather than fixed into four 2048-token partitions; four full 8k sequences would still require capacity for about 32k tokens. --kv-unified-per-slot can set an independent per-slot cap.⁷

vLLM: one paged KV pool, continuous batching

vLLM keeps a paged KV pool and allocates blocks on demand; llama-server can also use a unified pool, but its allocator and scheduling differ. Requests join and leave the batch at every decode step. Raising concurrency costs nothing when the server is idle because no per-request context is pre-reserved: a request's KV cache grows token-by-token in physical blocks, freed immediately when the request finishes.⁵ --max-num-seqs limits batch size; the true limit on concurrent requests is "how many sequences fit their accumulated KV into the pool."

Ollama: slots like llama.cpp, sized by defaults

Ollama inherits the slot model: OLLAMA_NUM_PARALLEL (default 1) is the number of parallel requests each model processes, and required RAM scales by OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH.⁸ With the default of 1, concurrent requests queue; each additional parallel request can raise KV memory demand.

GGUF: native home vs experimental guest

GGUF is llama.cpp's native format — the reference implementation, with the full quant family (Q2_K through Q8_0, i-quants, imatrix calibration).⁹

vLLM's GGUF support is a different story, in its own docs' words: "GGUF support in vLLM is highly experimental and under-optimized at the moment, it might be incompatible with other features." GGUF loading has migrated out of core into the OOT vllm-gguf-plugin package (uv pip install vllm-gguf-plugin), and the docs recommend passing the base model's tokenizer explicitly, e.g.¹⁰:

vllm serve unsloth/Qwen3-0.6B-GGUF:Q4_K_M --tokenizer Qwen/Qwen3-0.6B

Practical rule: on vLLM, GGUF support remains less optimized in vLLM, so performance needs measurement. For high-throughput serving, also compare safetensors with supported quantization schemes (AWQ, GPTQ, FP8).

Defaults that bite

Ollama context defaults are VRAM-tiered. Current docs: < 24 GiB VRAM → 4k context; 24–48 GiB → 32k; ≥ 48 GiB → 256k, overridable with OLLAMA_CONTEXT_LENGTH (older releases used a flat 4096 default, and 4096 is still what the FAQ's override examples show).¹¹ If your 24 GB card picks a 32k default you didn't ask for, KV cache grows accordingly — check ollama ps (its CONTEXT column) and set num_ctx per request or the env var globally.

Ollama KV cache type defaults to f16. OLLAMA_KV_CACHE_TYPE accepts f16 (default), q8_0 (~half the KV payload memory), or q4_0 (~quarter the payload; both need quality checks on your workload) — and requires flash attention to be enabled to take effect.⁸

OLLAMA_NUM_PARALLEL defaults to 1 — simultaneous requests queue rather than run concurrently, and raising it multiplies KV memory by N.⁸

vLLM gpu_memory_utilization defaults to 0.92 — vLLM claims up to 92% of the card's total memory at startup (per-instance, not per-process-coordinated), then subtracts weights, activation peak, and CUDA-graph pools; the KV cache is the residual. On a card shared with a display or other processes, this may leave too little free memory. Adjust the fraction or --max-model-len based on measured use.¹²

Worked example: 14B Q4_K_M, 4 slots × 8k, on a 24 GB card

Take Qwen2.5-14B (48 layers, GQA with 8 KV heads, hidden size 5120 across 40 heads → head dim 128, 14.7B total parameters).¹³ We compute the budget using the site's KV formula (see the KV cache guide and inference math), plugging in llama.cpp's measured Q4_K_M ≈ 4.85 bits-per-weight for this weight class.⁹

Weights at 4.85 bpw:

KV cache per token (f16, 2 bytes/value), using $2 \times L \times H_{KV} \times d_{head} \times b_{dtype} \times s$:

Per slot at 8192 tokens, and total for 4 slots:

It fits — with room to spare, which is the point: on a 24 GB card a 14B Q4_K_M with 4 concurrent 8k conversations is comfortably inside budget, and the ~8.65 GB headroom — 24 decimal GB on the sticker minus the 14.30 GiB subtotal (15.35 decimal GB) — covers compute scratch (dependent on batch size and backend) plus the CUDA context. Two levers if you need more slots or context: -ctk q8_0 -ctv q8_0 halves KV to 3.0 GiB (llama-server flags, f16 default¹), the same trade Ollama's OLLAMA_KV_CACHE_TYPE=q8_0 makes. For a full sizing workflow see the VRAM calculator explained and quantization guide; the interactive tool is at LLM inference calculator.

For four fixed 8k slots in non-unified mode, start llama-server -m model-Q4_K_M.gguf -c 32768 -np 4 --no-kv-unified. A unified pool can also support them if its total capacity is sufficient.

Decision table

Rule of thumb: llama.cpp when you need to touch the knobs or fit weird hardware; Ollama when you want zero-friction local use; vLLM when other people or services are hitting the endpoint and throughput per GPU matters.