vLLM vs SGLang: Architecture, Overhead, and When to Pick Which

Two open-source engines dominate self-hosted LLM serving in 2026: vLLM and SGLang. Both give you continuous batching, a paged KV cache, OpenAI-compatible HTTP endpoints, and pluggable attention kernels — yet they differ in where they spend their engineering effort, and those differences show up in your latency and throughput numbers. This guide compares both at the level that actually matters for a deployment decision: memory layout, scheduler design, CPU overhead, and feature support, each claim tied to a cited benchmark or official doc.

The shared DNA

Both engines inherit the same two load-bearing ideas, so the baseline feature set is nearly identical:

• Paged KV cache management. vLLM introduced this as PagedAttention (arXiv:2309.06180, SOSP 2023): KV cache memory is split into fixed-size blocks, allocated on demand like OS pages, which cut memory waste from fragmentation and duplication to near zero and improved throughput 2–4× over FasterTransformer and Orca at equal latency ¹. SGLang adopted the same block-level paging for its KV cache (page size 1 by default) — the radix tree sits on top of it.
• Continuous batching (iteration-level scheduling): requests join and leave the running batch between forward passes, instead of waiting for a whole batch to finish. SGLang's v0.4 blog explicitly keeps "batch scheduling, memory allocation, and prefix matching" in its CPU-side scheduler alongside GPUs running steps continuously ².
• OpenAI-compatible APIs. Both ship /v1/completions and /v1/chat/completions (vLLM documented on its online-serving page ³; SGLang reachable through the same OpenAI SDK, with structured-output examples in its docs ²).

If you have read the site's KV-cache deep dive, the memory numbers below will look familiar — both engines live and die by that one tensor.

A 60-second worked example (Qwen3-VL-32B, per this site's reference model)

For a GQA model with 64 layers, 8 KV heads, and head dimension 128, one token costs:

KV bytes/token = 2 (K and V) × 64 layers × 8 heads × 128 dim × 2 B (fp16)
               = 262,144 B = 256 KiB per token

A single 4,096-token context therefore pins 1.07 GB of KV cache in fp16 — either engine allocates this block-by-block, whether the pages come from a vLLM block table or an SGLang radix tree. The difference is purely in how reuse across requests is organized. Use the LLM inference calculator to plug in your own model; the math behind it is identical for both engines.

Pluggable attention backends — on both sides

Neither engine hard-codes its attention kernel:

• vLLM (V1) selects among FlashAttention (FA2/FA3/FA4), FlashInfer (including TRTLLM-style kernels), and Triton backends via a registry with per-platform priority; users can override with VLLM_ATTENTION_BACKEND ⁴. The V1 blog specifically credits FlashAttention 3 integration as a core piece ⁵.
• SGLang takes --attention-backend (e.g. triton, flashinfer); its saved v0.4 scheduler profile was itself captured on the Triton backend, with a note that the FlashInfer path still had a small idle-time gap at the time ². The default grammar/xgrammar backend and mm_attention_backend are separately configurable in server args ⁶.

So "engine X is faster" is almost never a property of the engine alone — it is engine × kernel × quantization × hardware × workload. Which is why every perf claim below cites its benchmark.

Team vLLM: PagedAttention, and the V1 rewrite

vLLM's origin is the PagedAttention paper: virtual-memory-style block paging for the KV cache, achieving near-zero cache waste, flexible cross-request cache sharing, and 2–4× throughput over the then state of the art (FasterTransformer, Orca), with the gap widening for longer sequences and more complex decoding ¹.

The bigger architectural story today is the V1 rewrite (announced January 2025). The design goals were a modular codebase, near-zero CPU overhead, combined optimizations, and "zero configs" — features on by default ⁵. Concretely:

• Up to 1.7× higher throughput than V0 (without multi-step scheduling), attributed to CPU-overhead reductions across the stack — the GPU kernels were essentially identical between V0 and V1, so the gain is scheduler/execution-loop engineering, measured on Llama 3.1 8B and Llama 3.3 70B with ShareGPT traces, and larger still on Qwen2-VL (VisionArena traces) ⁵.
• Zero-overhead prefix caching, on by default. V0's prefix caching cost significant CPU and was disabled by default; V1's reworked data structures keep the penalty under 1% throughput at 0% cache hit rate, so it ships enabled ⁵.
• torch.compile + piecewise CUDA graphs to squeeze kernel launch and Python overhead without hand-writing kernels for every model ⁵.
• Symmetric TP architecture: the scheduler and all workers (including worker 0) run as separate processes communicating incremental state diffs, removing V0's colocated scheduler/worker-0 special case ⁵.
• Multimodal as a first-class citizen: image preprocessing (decode/crop/transform) moved to a separate non-blocking process with a preprocessing cache, image-hash prefix caching for VLM multi-turn traffic, and an "encoder cache" enabling chunked prefill for multimodal inputs ⁵. VLM gains were the largest in the V1 benchmarks ⁵.

Since the V1 alpha, the engine has become vLLM's default code path and the feature list has kept growing; the docs today cover speculative decoding (draft models, EAGLE, MTP, n-gram), LoRA, structured outputs, and quantization as first-class features ^{7 8}.

Team SGLang: RadixAttention, and the overlap scheduler

SGLang's identity is RadixAttention (arXiv:2312.07104): all past KV caches live in an LRU-managed radix tree, and any new request automatically reuses the longest matching cached prefix — no configuration, no collision-prone hashing, no manual cache warming. The paper reports up to 6.4× higher throughput vs. contemporary inference systems on prefix-heavy workloads (few-shot, agent control, RAG, multi-turn chat, JSON decoding) ⁹. We deliberately don't re-derive the tree mechanics here — see the site's KV-cache article for exactly how radix-tree reuse works and when it degrades to vLLM-style block reuse.

The second pillar is the zero-overhead (overlap) batch scheduler, shipped default-on in v0.4 (December 2024): the CPU scheduler runs one batch ahead of the GPU, preparing the next batch's metadata while the current one computes, hiding radix-cache operations and other CPU work. An Nsight profile of 5 consecutive decode batches shows no GPU idle time (captured on the Triton attention backend; a minor gap remained on FlashInfer at the time). Result: 1.1× vs SGLang v0.3 and 1.3× vs other state-of-the-art baselines, with gains "most significant on small models and large tensor parallelism sizes" ².

The honest caveat: the overlap win is not universal. Issue #2558 on the SGLang tracker reports Qwen2.5-0.5B on a single A100 measuring faster with --disable-overlap, reproducibly, on the same bench_serving random workload (4,096-token input, 2,048-token output) the v0.4 blog used ¹⁰. The v0.4 claim is specifically "small models and large TP sizes" ² — with a tiny model on one GPU, forward steps can be shorter than the bookkeeping the overlap pipeline adds, and the fix direction discussed in the issue is run-to-completion scheduling for small models ¹⁰. Read the two sources together: overlap helps most when GPU steps are long relative to scheduler CPU work; benchmark your own small-model fleet rather than assuming default-on means always-faster.

Three more v0.4 numbers worth knowing:

• Cache-aware load balancer (sglang-router, Rust): routes each request to the worker with the highest predicted prefix hit rate. On a balanced shared-prefix workload across 8× A100: throughput 82,665 → 158,596 tok/s (≈1.9×), cache hit rate 20% → 75% (3.8×) ².
• DP attention for MLA models (DeepSeek): each DP rank owns its own KV cache, eliminating TP's KV duplication for single-KV-head MLA, all-gathering before the MoE layer — 1.9× decoding throughput on DeepSeek-Coder-V2 across 8× H100 (at TP 8) ².
• xgrammar structured outputs: up to 10× faster JSON decoding than other open-source engines, via adaptive token masks over the grammar's context-free kernel ^{2 11}.

Later SGLang work added EAGLE-2/EAGLE-3 speculative decoding (2.36× throughput on Llama 3.1 8B on MT-Bench on one H100 in the docs' table: 158.34 → 373.25 tok/s with EAGLE-3) ¹², and S-LoRA/Punica-derived multi-LoRA support ⁶.

Differences users actually feel at the keyboard

Beyond the headline benchmarks, four areas decide most real deployments. All entries below are from the projects' current official docs.

1. Prefix-cache ergonomics

vLLM V1's hash-based APC is now default-on and near-free, but reuse is exact-prefix and block-granular. SGLang's radix tree is automatic, sub-block granular, and comes with fleet tooling around it: lpm scheduling (group requests by longest shared prefix), --disable-radix-cache for ablation, and the cache-aware router. For multi-tenant chat with heavy system-prompt reuse, SGLang's ecosystem is ahead; for a single node with diverse traffic, it's a wash.

2. Speculative decoding

vLLM supports draft-model speculation, EAGLE, MTP (multi-token prediction), n-gram/lookup speculation, and dynamic speculation, with documented algorithmic-losslessness validation and one sharp edge: pipeline parallelism is incompatible with speculative decoding as of vLLM ≤ 0.15.0 ⁷. SGLang's flagship is EAGLE-2/EAGLE-3 (plus standalone draft models and MTP for newer architectures), compatible with radix cache and chunked prefill ¹². Historically each engine has led on out-of-the-box spec-decode throughput for different draft-model families in different weeks — benchmark with your draft model pair, not with either project's chart.

3. Quantization support

vLLM's quantization page lists AutoAWQ, GPTQModel, BitsAndBytes, Intel Neural Compressor, LLM Compressor (FP8 W8A8, INT4 W4A16, INT8 W4A8/W8A8), NVIDIA Model Optimizer (incl. NVFP4-class recipes), online quantization, AMD Quark, TorchAO, GGUF, and quantized KV cache ¹³. SGLang's table covers fp8, w8a8 fp8/int8, blockwise_int8, mxfp4/mxfp8 variants, awq, gptq (via gptq_marlin on NVIDIA/AMD), compressed-tensors, quark, and auto-round, across NVIDIA/AMD/Ascend ¹⁴. Practical overlap is huge (FP8, AWQ, GPTQ-Marlin, INT8, MXFP4 both sides); differences are mostly in niche format/hardware corners — check your specific checkpoint's quant recipe against both tables before committing.

4. LoRA and parallelism

Multi-LoRA: both support serving many adapters in one batch. vLLM exposes --enable-lora, --max-lora-rank (default 64 for the OpenAI server), plus runtime dynamic LoRA loading via API (opt-in, security warning) ⁸. SGLang's multi-LoRA is S-LoRA/Punica-derived (--lora-paths, max_loras_per_batch default 8, Triton and chunked-SGMV backends, LRU adapter eviction, TP-compatible) ⁶.

Distributed: vLLM runs tensor, pipeline, expert, and data parallelism (with --enable-expert-parallel for MoE) ^{8 7}. SGLang covers TP, PP, EP, DP attention (--enable-dp-attention, the MLA path above) and ships its own cache-aware, multi-node-capable router ^{2 12}. For DeepSeek-style MLA architectures at large TP, SGLang's DP-attention story is the more differentiated one; for heterogeneous fleets, vLLM's hardware matrix (NVIDIA, AMD, TPU, CPU) is broader ¹³.

Decision table

TensorRT-LLM, for completeness

NVIDIA's TensorRT-LLM¹⁵ is the third name in every one of these comparisons. It has also been moving toward the same pattern: the current stack centers on a PyTorch backend (available from v0.17 onward, tensorrt_llm._torch) plus trtllm-serve with an OpenAI-compatible API, overlapping heavily in feature set with the two open engines — continuous batching, in-flight updates, FP8/NVFP4 quantization, and the SPECULATE family of speculative-decoding methods remain its hallmarks, at the cost of a tighter model/engine-version coupling and a mostly-NVIDIA-only hardware matrix. In exchange you get NVIDIA-tuned kernels and, on supported stacks, the highest absolute numbers the vendor publishes. A fair rule from the outside: if your fleet is homogeneous NVIDIA and you can accept the release cadence, benchmark it as the proprietary baseline against whatever wins the vLLM/SGLang decision above; if you need engine-agnostic artifacts, cross-vendor GPUs, or maximum hackability, the two open engines are the practical choice. The vLLM V1 authors themselves list TensorRT-LLM among the engines that influenced V1's design ⁵.

Bottom line

vLLM and SGLang converge on the fundamentals — paged KV, continuous batching, OpenAI APIs, pluggable attention — and compete on the edges. vLLM bet on a ground-up execution-loop rewrite (V1: 1.7× over V0, prefix caching free-and-on, multimodal-first); SGLang bet on automatic prefix reuse everywhere (radix tree + cache-aware router) and aggressive scheduler overlap. Your workload decides: reuse patterns and routing favor SGLang; breadth, VLMs, and hardware reach favor vLLM. Either way, never deploy on a headline number — rerun bench_serving-style traces on your own traffic, on your own GPUs, with quantization and context lengths you actually ship. If you're new to the underlying mechanics, start with how LLMs work and the VRAM model behind these numbers.