Skip to content
Friendly disclaimer: flozi00 TechHub is a solo side-project next to a full-time job — personal learning notes, no official statements. Verify critical steps yourself.
flozi00 Logoflozi00 TechHub
ENDE

How we serve GLM on B300 in production: MTP k=5 tuning

Production notes from serving a GLM-class MoE on 8x B300: the acceptance-rate math behind speculative decoding, why k=2 captures most of k=5 ceiling, and the NVLS failure mode that cost us a week.

3 min readflozi00
llm-inferencespeculative-decodingmtpb300ncclproduction

Multi-token prediction in production is an acceptance-economics problem. This post is our tuning notes from serving a GLM-class MoE on 8x B300 (SM120): what speculative depth buys once the compounding acceptance math is applied, and the NVLink-SHARP (NVLS) issue that cost a week to diagnose.

The setup

GLM MTP head ships inside the checkpoint - one extra prediction layer, not a separate draft-model download. LMSYS documents the GLM-4.5 architecture with its native MTP head for speculative decoding1, and vLLM (speculative_config={"method": "mtp", "num_speculative_tokens": k}) and SGLang (--speculative-algorithm NEXTN) consume it directly2.

The acceptance math that predicts your speedup

Per-position acceptance rates are the only numbers that matter. Expected tokens per decoding step is not k plus one - it is driven by the chain probability of joint acceptance:

E[tokens per step]=1+∑i=1k∏j=1iaj\mathbb{E}[\text{tokens per step}] = 1 + \sum_{i=1}^{k} \prod_{j=1}^{i} a_j

With per-position acceptance of 85%, 67%, 51%, 36%, 31% (T+1..T+5, community-measured on GLM quantizations on vLLM), the chain compounds brutally:

kjoint chainE[accepted]tokens/stepshare of k=5
10.850.851.8563%
20.85, 0.571.422.4277%
30.85, 0.57, 0.291.712.7192%
40.85, 0.57, 0.29, 0.101.812.8197%
50.85, 0.57, 0.29, 0.10, 0.0321.852.85100%

So k=2 already yields 1.42 of the ultimate 1.85 accepted tokens - k=2 captures roughly 77% of the k=5 ceiling, and k=3 captures ~92%. Deeper speculation buys marginal tokens while every draft position adds cost:

  • Wasted verify work: a rejected draft still consumed a verify slot.
  • Rejection latency: the corrected token must propagate before the next draft starts - deeper chains stall longer per rejection.
  • KV-cache pressure: num_speculative_tokens reserves KV entries for draft positions per request; vLLM docs recommend starting small2.

Our production take: structured-output workloads (predictable grammar) accept deeper chains and tolerate higher k; prose does not. Depth is a per-workload decision driven by measured acceptance - measure first, then set k.

The lost week: NVLS on SM120

The second story is a fabric problem. During bring-up, node-local all-reduce ran far slower than the bandwidth math predicts. Root cause: the collective silently fell back off NVLink SHARP (NVLS) - and finding that the failure was silent cost roughly a week.

The mechanism, once found:

  • NVLS is NVLink SHARP: in-network reduction offloaded into the NVSwitch domain; NCCL supports it since 2.17 on third-gen NVSwitch with Hopper or later3.
  • Its default is silent: NCCL_NVLS_ENABLE defaults to 2 (auto-detect). If NVLink SHARP resources cannot be allocated, NCCL falls back to a slower algorithm without an error; NCCL 2.27.3 added graceful fallback, with NCCL_NVLS_ENABLE=1 preserving the old loud-failure behavior4.
  • Driver and fabric-manager mismatch was the trigger: packages that were each fine individually but mismatched together made NVLS init fail silently. The collective still ran - just slower. The symptom was throughput off by a factor, not a crash.

The working rules we kept:

  1. Pin driver + fabric-manager + NCCL as a set, not three versions.
  2. Probe with a tiny all-reduce benchmark before blaming the serving stack.
  3. NCCL_DEBUG=INFO names the algorithm that actually ran - trust the measured probe over assumed topology.

Transferable rule: on new silicon, verify the fast path is active - silence is the fabric failure mode.

Notes on output fidelity

Speculative decoding with verifier acceptance changes throughput, not the output distribution: accepted tokens are exactly the tokens the trunk model would have produced, so quality is unchanged by construction - for MTP heads the same as for separate EAGLE draft models.

Footnotes

  1. LMSYS, "GLM-4.5 Meets SGLang" - GLM-4.5 architecture (native MTP head) and vLLM/SGLang speculative flags: https://lmsys.org/blog/2025-07-31-glm4-5 ↩

  2. vLLM MTP documentation (speculative_config, num_speculative_tokens): https://docs.vllm.ai/en/latest/features/speculative_decoding/mtp/ ↩ ↩2

  3. NVIDIA NCCL user guide - NCCL_NVLS_ENABLE (default 2, auto-detect): https://docs.nvidia.com/deeplearning/nccl/archives/nccl_2283/user-guide/docs/env.html ↩

  4. NCCL 2.27.3 release notes - graceful NVLS fallback: https://docs.nvidia.com/deeplearning/nccl/release-notes/rel_2-27-3.html ↩