Serving Qwen3.8 on MI350P: Kernels, Failures, and Measurements

The useful result of this optimization campaign is a set of concrete changes to an inference dataflow. Qwen3.8-Flash-Next moved from 131.6 / 702.4 / 1,168.2 aggregate output tokens/s in the first working AMD configuration to 282.0 / 1,199.2 / 2,256.6 in configuration k38, at 1, 8 and 32 concurrent streams. That is +114.4%, +70.7% and +93.2% on the same AMD deployment path. The work reduced repeated recurrent-state writes, reused attention keys across verification rows, fused routing and quantization, and changed the precision of sensitive residual projections.¹²

These are first-party observations from my serving campaign, rather than an independent hardware ranking. The accompanying CSVs contain 118 concurrency-level observations across 40 configuration labels, dated 8–10 October 2026. The public plugin source makes the mechanisms inspectable. This write-up recomputes the comparisons, preserves invalid runs and regressions, and distinguishes code, measurements and explanations. It does not rerun the GPU experiments.

The main figure uses one complete NVIDIA recheck and one complete AMD configuration. Their hardware, parallelism and quantization differ. Each panel has its own zero-based scale. Open the vector figure to inspect the values.

What was actually compared

The AMD deployment used one MI350P, 144 GB HBM3e, 128 compute units, TP1 and a custom MXFP4 checkpoint. The NVIDIA deployment used two RTX PRO 6000 Blackwell Server Edition cards, each with 96 GB GDDR7, TP2 and NVFP4. Both sides had custom kernel work; the NVIDIA reference was a production deployment, not an unoptimized control.³⁴

The campaign notes identify NVIDIA work around vLLM 0.30.0 and an AMD image containing vLLM 0.31.1rc1.dev88, ROCm 7.2.3 and AITER 0.1.24.post1. Consequently, “same vLLM” would be too strong: these are distinct backend builds and precision recipes. The inspected plugin revision is 8d09f300f4dc1542081c39abe7521f1e29508f5a. The CSV records endpoint aliases, not weight hashes, image digests or complete launch manifests.

The reported AMD checkpoint occupied about 115 GiB. That exceeds one 96 GB card's capacity. A single larger-memory GPU can avoid the tensor-parallel collectives that a two-card deployment requires, but this comparison does not isolate that effect. It also does not establish energy efficiency or total cost: there are no synchronized power or cost records in the ledger. See the inference math guide for the separate roles of compute, memory traffic and communication.

Audit the benchmark before interpreting the speedup

The archived abbench.py runs closed-loop streams: each starts its next request after the previous request completes. It asks for 384 output tokens, uses temperature 0.2, sets ignore_eos=True, and counts returned token IDs rather than counting streaming events. The report describes essay prompts and nominal 60-second windows; the ledger also includes longer-window and profiler-labelled runs.⁵¹

Aggregate throughput is total successful output tokens divided by measured wall time. Workers stop starting requests at the deadline, then finish their current request. The wall interval therefore includes that drain. A per-request median decode rate excludes initial startup time and uses a different denominator. Multiplying that median by concurrency does not reconstruct aggregate throughput. The prompt generator uses time-dependent seeds, so separate arms share a prompt-generation recipe rather than an identical replay corpus.

The report says there was a 20-second warm-up per load level. The archived harness itself only sends one 32-token warm-up request, and the CSV has no warm-up metadata. Additional manual warm-up may have occurred, but it cannot be verified from these artifacts. The reported thermal clocks, long-duration degradation and post-cutover tests likewise have no corresponding telemetry or rows here. They remain campaign notes, not measurements redrawn from the CSV.

The original headline used the best NVIDIA throughput per load level: 184.7 / 776.9 / 1,627.7, combining two different sweeps. The complete 9 October recheck instead reads 184.7 / 776.9 / 1,533.1. Against that complete sweep, k38 is +52.7%, +54.4% and +47.2%. Against the larger observed NVIDIA C32 value, its C32 advantage is 38.6%. Both calculations are valid descriptions of their denominators; they are not interchangeable.

Likewise, 282.0 / 1,206.2 / 2,405.1 is not one AMD configuration: it selects C1 from k38, C8 from k41 and C32 from k39. The article uses k38 as its main result and names the other observations explicitly. The two NVIDIA sweeps and the many evolving AMD arms are too few and too heterogeneous to manufacture confidence intervals or attribute every change causally.

The report's older NVIDIA “stock” row, 198 / 711 / 854, used a different repetition-corpus harness and has no rows in the supplied October ledger. It is excluded from these figures. Combining it with the three comparable-harness rows would not create a controlled four-way stock/optimized hardware comparison.

The time axis starts at the first AMD observation. CSV timestamps carry no timezone. The grey band is the range of two NVIDIA observations, not an uncertainty interval; the crosses include runs that must not count as correct throughput. Open the full history.

Why this model exposed several different bottlenecks

The pinned upstream configuration has 48 layers: 36 Gated DeltaNet (GDN) and 12 Qwen Sparse Attention (QSA). Residual width is 2,560, with four gated residual branches and a low-rank width of 320. Every layer has 512 routed experts, ten selected per token, and one shared expert with an intermediate width of 640. Four-token speculative drafting produces up to five target-verification positions per request, so C32 can present 160 rows to a small decode kernel.⁶

The work is not uniform. GDN repeatedly updates an FP32 recurrent matrix. QSA scores and gathers selected context. MoE dispatches small token groups into expert GEMMs. The four residual branches introduce dependent projections and gates. A useful optimization must match the tensor shape and taken backend path; a model-level “6B active” descriptor cannot predict these costs. The architecture guide explains the n-gram table and residual design in more detail.

Lesson 1: separate a valid baseline from a fast baseline

The first AMD point already includes a custom quantized checkpoint and correctness patches. Calling it “stock vLLM” would hide prerequisites. The public patch code describes three distinct incidents: a hostcall path rejected on a platform without PCIe atomics, redundant MoE padding, and reads of uninitialized data in a tuned FlyDSL intermediate layout.⁷

The padding case has a concrete producer/consumer contract. The logical expert intermediate width is 640, while the selected layout uses 768 columns. If stage 1 writes only the logical columns and stage 2 consumes the padded width, the untouched 128 columns and their scale bytes must contain valid zeros. An E8M0 scale byte of 0xFF denotes NaN. A recycled allocation can therefore turn a fast path into silent corruption. The plugin uses zero-initialized persistent buffers to retain the invariant without two per-call fills per layer. Buffer reuse also requires compatible layout and non-overlapping lifetime; “persistent” alone does not make it safe.

The linked upstream padding PR concerns GPT-OSS and documents related alignment problems. It is corroboration for the class of defect, not independent verification of every Qwen result in this campaign. The source rewrite fails when its expected anchor changes instead of silently enabling an unaudited backend. That revision check matters when the serving engine is changing weekly.⁸

Lesson 2: store fewer recurrent snapshots and replay the accepted prefix

For one value head, write the GDN state as a matrix $S_t\in\mathbb{R}^{128\times128}$. In the implementation's orientation, a normalized key $k_t$ addresses columns and a value $v_t$ supplies rows. A simplified recurrence, with decay $\alpha_t$ and update gate $\beta_t$, is:

Here $q_t$ includes the implementation's query normalization/scaling. The source computes the decay from its softplus gate and updates each value-head matrix. With 48 value heads and FP32 storage, one request/layer snapshot contains 3 MiB. Ordinary MTP-4 verification writes five such snapshots so the next step can choose the one corresponding to the accepted prefix.⁹

Deferred commit stores the state after token 0, which is always accepted, and compact inputs for the remaining tokens. The next step reconstructs the accepted state by replaying only its accepted prefix. It trades repeated full-matrix writes for compact records and extra recurrence work. Prefix-cache alignment is part of the contract: near a boundary the implementation writes the ordinary snapshots so other cache readers receive the expected state.

The byte calculation must use the whole stack. Under the stated full-write assumption:

One full write per target layer/request would be 3.375 GiB, plus records and boundary fallbacks, saving up to 13.5 GiB of full-state writes in that logical comparison. The original report's 4.6 GiB estimate does not follow from its stated dimensions. These numbers count logical payload, not hardware-counter HBM traffic, total allocation or measured step time. Cache behavior and replay arithmetic still matter.

A later variant expresses blocks of recurrence work with FP32 matrix-core operations. The reported C32 layer time changed from 181 to 127.6 µs: 29.5% less time, or 1.42× isolated throughput. Those are kernel observations reported by the campaign, not an end-to-end 42% speedup. The scalar replay variant checks bitwise state equivalence; the matrix-core variant has a different accumulation order and checks tolerance against an FP64 recurrence. The code makes that distinction explicit.

Lesson 3: a sparse operator can have an exact dense special case

QSA selects a budget of 2,048 context tokens through compressed-block indexing. When all visible tokens fit in that budget, the selected set can be the entire causal prefix. Gathering a sparse list then adds indirection without dropping any data. The dense path traverses the page table in token order and shares K/V tiles across a request's verification queries.¹⁰

For this checkpoint, each KV head serves 12 query heads. Five verification positions therefore form 60 query rows per KV head, fitting the kernel's 64-row tile. The optimization reuses each K/V tile across those queries, applies each row's causal mask and merges partial softmax statistics. It does not replace long-context sparse selection with arbitrary dense attention. Eligibility includes sequence length at most 2,048 and tile capacity; draft steps reusing an earlier sparse selection bypass the dense path. Below four requests, the code also retains the stock path.

The reported QSA work at C32 fell from about 2.85 to 0.7 ms per step in the campaign's profile. That is a module measurement with a specific eligibility pattern. It is neither a fourfold model speedup nor proof of a universal long-context win. The token set is equivalent under the guard, but token order and reduction order can change BF16 rounding.

Lesson 4: optimize dispatch and precision at the actual decode shapes

The routed feed-forward input is [M, 2560]. Routing produces ten expert IDs and weights plus a shared-expert slot, giving [M, 11] dispatch metadata over 513 expert entries. The fused path computes gating and token-row MXFP4 quantization together, then builds the expert-sorted layout from bitmaps in a second launch. It is intended for small $M$, up to 160 in this preset; it does not claim to replace every large-prefill path.¹¹

Its exactness contract includes lower-index tie handling, distinct expert IDs even under non-finite warm-up logits, matching exponential arithmetic and padding stores. The reported routing times at $M=5/40/64$ fell from 21.7/30.3/33.0 to 11.7/14.1/14.5 µs. Removing launches is useful only if the replacement's registers, memory layout and occupancy still fit the device. Tactic tables tuned for a 256-CU accelerator are not automatically suitable for the 128-CU MI350P.

Precision is another local decision. The residual projections operate on the four-branch width 10,240, through a 320-wide intermediate. The campaign rejected MXFP4 for those projections after large logit deviations and adopted FP8 W8A16 instead. The source stores per-output-channel scales, widens FP8 weights to BF16 in the kernel and scales the accumulator. This is an approximate weight path, not a claim that all model arithmetic remains bitwise unchanged.¹²

The vocabulary projection is also bandwidth-heavy: [248320, 2560] at BF16 contains about 1.27 GB decimal of weights. Its MXFP4 screen rescores the top 64 candidates using original BF16 rows while leaving approximate logits in the tail. Rescoring a candidate is exact for that row; it does not make the full sampling distribution identical, nor guarantee that every true top candidate was screened in. The randomized greedy oracle is a regression check, not a mathematical completeness proof.¹³

For skinny GEMMs, launch geometry is inspectable. The basic split-K grid count is $\lceil M/B_M\rceil\lceil N/B_N\rceil S$. With $M\le16$, $B_M=B_N=128$, $N=5120$ and $S=5$, that is 200 blocks, versus 40 without split-K. An 800-block figure requires another tile change; split-K five alone does not produce it. More blocks can improve occupancy, but temporary partials and the reduction add memory traffic. A faster GEMM therefore needs measurement at the actual $(M,N,K)$ and cache state.

Lesson 5: keep regressions in the decision record

The campaign's most informative rejected ideas explain where simple intuitions failed. The timings below are reported microbenchmarks; they are not individually recoverable from the aggregate CSV.⁴⁹

The k39 observation gains 6.6% throughput over k38 and has 2.63× the maximum TTFT. The maximum is not a percentile estimate and depends on sample count: these C32 observations have 398 and 378 successful requests respectively. The later k41 run has 2,326.1 tokens/s but a 4.887-second maximum TTFT. It supports a throughput observation, not an unconditional claim that every latency metric improved. Open the tradeoff figure.

The ledger's itl_p99_ms also needs a precise name. The harness divides a streaming chunk's gap by the number of token IDs in that chunk and inserts that value once per token. Speculative decoding can deliver several IDs simultaneously. This is a burst-normalized chunk-gap p99, not the true arrival gap of every token. k38 reports 9.74/19.07/40.75 ms versus the NVIDIA recheck's 15.15/28.48/61.33 ms. It is a useful same-harness proxy, but it can hide the duration of a multi-token burst gap and does not alone establish smoother delivery.

What correctness and production validation establish

The public source includes boot gates and multi-step numerical oracles, covering accepted-prefix lengths, cache-boundary copies and graph replay. Those are valuable mechanism checks. Approximate quantization and changed reductions still need task-level evaluation, and a finite passing oracle cannot prove correctness for every possible input.

Campaign notes report full GSM8K scores of 0.9689 for AMD k38 and 0.9674 for NVIDIA, plus tool, needle and image checks. The nominal difference is about two questions out of 1,319. Per-example predictions and evaluation manifests are absent from the supplied CSVs, so there is no paired significance or equivalence result here. The post-cutover score 0.9583 is also below the notes' stated 0.9606–0.9689 iteration range. It should be investigated, rather than declared inside that range or dismissed as numerical noise. “No HTTP errors,” “oracle pass” and “quality parity” answer different questions.

The notes record a production cutover on 11 October and zero failed probes. They do not provide a request-level availability log, so zero failed probes must not become zero failed production requests. Thermal degradation is another reported observation that needs a steady-state experiment: the available short sweeps contain no time-aligned temperature, clock or power samples. A cooled long-duration A/B test remains necessary before treating these numbers as sustained capacity.

For the next run, the useful measurement record is concrete: checkpoint and image hashes, full flags and gate settings, replayable prompts, actual warm-up and drain times, prompt/output lengths, draft acceptance, streaming burst sizes, request errors, quality outputs, and time-aligned thermal/power telemetry. Alternate backend order and repeat complete sweeps. Record the taken kernel path at startup, and retain an ordinary path for shapes outside each optimization's verified contract.

The transferable experience is to follow the bytes and dependencies. Defer states that are rarely consumed, reuse K/V when the selected set permits it, fuse metadata only within an exact contract, and choose precision per projection. Keep the baseline valid and the latency distribution visible. The observed AMD gains are substantial for this deployment; their value comes from the mechanism and the auditable measurements, not from extrapolating a vendor-wide result.

Data and primary sources

The full ledger, configuration ladder, audit output and reported context accompany the article. The two CSVs preserve the supplied bytes. scripts/plot-mi350p-campaign.py validates the joins, computes comparisons and regenerates all four figures in both languages with Matplotlib 3.10.8. The source revision links below describe the implementation; its reported timings were not independently rerun for this article.