Who Pays for the KV Cache? Metering Rules Are a Pricing Decision in Disguise

Every FinOps conversation about self-hosted inference eventually collides with a question the public clouds answer with a price sheet and the private ones answer with a fight: when four tenants share one GPU and one KV cache, who pays? Some API providers publish cache-read and cache-write rates; those price cards make the allocation policy visible, as discussed in the cache-read price comparison. On a Kubernetes cluster running your own vLLM, nobody hands you a metering policy. You pick one, and the pick silently re-prices every tenant.

That is the entire subject of arXiv:2609.24991, "Who Pays for the KV Cache? Attributing Shared AI Inference Spend Across Kubernetes and LLM Provider Bills" by Timothy Urista (submitted September 21, 2026)¹. The paper has two halves that are usually kept apart: a tool, unalloc, that joins OpenCost, LiteLLM, OpenAI and Anthropic cost data into one decimal-exact ledger and reports the share of spend with no owner; and a measurement, on an H100 running vLLM, of how much the metering rule alone — not the workload, not the hardware, just the billing key — moves money between tenants. We verified both halves at the primaries; this guide decomposes the second, because it is the one that generalizes. If you run agentic workloads with heavy prefix reuse, it is about your bill.

The headline, verified

On a DigitalOcean H100 80GB droplet (a 29-minute, approximately $2.15 run, fully captured in the repo), vLLM 0.29.0 served Qwen2.5-7B-Instruct in bf16 with an 8,192-token context and prefix caching; vLLM sized its KV pool at 995,296 tokens². Four tenants share the pod. Across configured loads of 2 to 16 requests per second — 3.7 to 26.9 completed requests per second, because the configured rate counts session-initial arrivals only — the two meters disagree at every load²:

The abstract rounds this to "12–14 percentage points at every load tested." Subtracting the displayed, rounded Table 3 shares gives 11.7, 11.9, 12.5 and 13.6 points; the paper's prose reports an upper end of 13.7 points from its underlying figures. The table does not disclose enough decimals to reconstruct that last unrounded gap². In these four runs the direction is consistent and the reported whole-run gap increases with configured load. Each load has just one two-minute run; the ten-second windows describe variation within it and provide no run-to-run uncertainty estimate. nvidia-smi reports 97–99% GPU utilization, but throughput increases about sevenfold across the settings: this utilization metric does not prove saturation or identify a fair cost allocation².

Also verified, and worth stating plainly: all 6,241 requests completed without error, and the served model stores about 57 KB of KV per token under the paper's own accounting².

Why the meters disagree in this run

The paper's synthetic traffic gives search requests long prompts (600–1,400 tokens) and short answers (16–64 tokens), while agents send multi-turn prompts and generate longer answers². A raw token meter counts prompt tokens even when vLLM reuses a cached prefix; an equal time-share meter divides each 50 ms among requests then in flight. On the measured H100 run, search receives 16.5–18.9% of the redistributed bill under tokens and 4.7–5.3% under time share. Those are different allocation rules, not evidence that either is the economically correct price. Their ordering is an observation for this workload and can differ with tenant mix, cache-hit rates, output lengths and batching².

KV occupancy introduces another resource and potentially another meter. The paper gives about 57 kB of KV per token and a 995,296-token pool. For scale only, suppose a session actually retains 32,768 KV tokens:

# Hypothetical retained context, using the paper's rounded 57 kB/token.
held_tokens = 32_768
pool_tokens = 995_296
kv_bytes_per_token = 57_000  # decimal kB; approximate paper figure
print(round(held_tokens / pool_tokens * 100, 2))       # 3.29% of token pool
print(round(held_tokens * kv_bytes_per_token / 1024**3, 2))  # ~1.74 GiB

This is a capacity example, not an observed paused session or a monthly bill. Billing KV residency would need block ownership and duration, cache-eviction behavior, a rate for reserved capacity and a rule for unused blocks. Neither the raw-token nor equal-time meter measures KV byte-seconds. In the H100 validation, observed KV-pool occupancy was only 0.7–8.1%, so this run does not show KV capacity as the binding resource².

Neither meter is ground truth — and the paper says so

The single most respectable sentence in the paper is that neither meter is a ground truth, with the results positioned against exact Shapley-based energy attribution work¹³. Equal time-share is itself a heuristic — it charges a request waiting on prefill the same as one decoding. List-price weighting (cached input at 0.1x, output at 4x) is no better: in the simulator it moves the agents tenant twelve points away from measured step time². Neither meter is a bound on a uniquely correct tenant share. The paper recommends choosing and publishing the meter and carrying idle GPUs, empty KV pools and communication as explicit overhead before redistributing them by policy². This is the on-prem mirror of the provider-side read-pricing fight: someone pays for cache residency; the only question on your cluster is whether you chose who, or let the join key choose.

The Kubernetes seams: excellent findings, synthetic clothes

The other half of the paper — where attribution breaks between Kubernetes allocations, gateway logs, and provider bills — is legitimately good failure-mode engineering, and the paper itself is scrupulous about its scope in a way the secondhand summaries are not. These are constructed cases: a month of synthetic OpenCost allocations for a multi-pod deployment, plus synthetic gateway/provider billing fixtures served through mock APIs. They are not observed production bills²:

• Labels set only on LeaderWorkerSet leader pods leave 66% of that deployment's GPU bill unowned (65.9% in the detailed run)².
• The natural fallback key drops the headline unallocated share to 4% (4.4%) — by routing $23,597, i.e. 61% of the bill, into a bucket called vllm, the Helm chart's app.kubernetes.io/name, which collides with the LeaderWorkerSet canonical key on name².
• Enabling every source double-counts all gateway spend¹.
• Reading one page of a billing API reports a quarter of spend¹.

Read the second bullet twice, because it is the real contribution: the fallback does not fix attribution, it converts unknown spend into wrongly-owned spend while making the headline metric look better. A tool that only reports an unallocated percentage cannot distinguish that scenario from a correct one; the paper's fix is reporting the fallback-attributed amount separately and making collision resolution order-independent².

The first two findings come from the synthetic OpenCost month; the latter two come from mocked billing APIs and synthetic spend. All four are fixture-based demonstrations with source and captured outputs in the repository²⁴. The communication example uses CPU-over-loopback PyTorch/Gloo timings to size a synthetic GPU-allocation month; it is not a measurement of NCCL traffic on GPUs. In that constructed case, $12,059 is attributed to collective communication and a per-token showback shifts $3,737 from the tensor-parallel tenant to the pipeline-parallel one. At the paper's more GPU-like 10% communication sensitivity setting, the shift is $166².

The ecosystem is moving the same direction

This is not a lone-wolf problem. OpenCost 1.121.0 (release tagged July 20, 2026; announced August 5, 2026 on the CNCF blog) shipped "AI Inference Costs v1" — first-of-a-kind Kubernetes inference cost tracking built on vLLM's vllm:prompt_tokens_total and vllm:generation_tokens_total metrics, validated on a 109-GPU, 30-model proof-of-concept cluster, exposing llm_total_hourly_cost and llm_cost_per_million_tokens through Prometheus and the REST API⁵⁶. The design distinction it makes is exactly this paper's thesis dressed as plumbing: allocation-based cost per million tokens is the full hosting cost — weights, GPU, and a share of gateway and KV-cache storage — while usage-based cost counts only infrastructure consumed during active inference, credits KV-cache-hit savings, and attributes cost to the tokens actually processed⁵. The gap between these model-level cost bases represents hosting overhead in the CNCF explanation; its example is usage-based $1.00/M versus allocation-based $4.00/M, corresponding to 25% utilization under that definition⁵. That is related to, but distinct from, the paper's tenant-share comparison. OpenCost PR #3845 listed workload and team attribution and detailed KV-cache optimization cost tracking as future work when the feature merged; this article does not establish their status in later releases⁷. unalloc itself is on PyPI at 0.2.3 — the release the paper's archive and fixtures are pinned to — joining the four major sources, with OpenCost's allocations as one input among several⁸¹.

The practical checklist, for anyone running a shared inference server today:

1. Decide the meter in writing — token share, time-share, or a hybrid — and put the decision in the showback itself, not the wiki.
2. Report fallback-attributed spend separately from unallocated spend. Never let a Helm chart name own a bill.
3. Publish both meters when tenants dispute. The 12-point spread is the size of the argument you will otherwise have.
4. If you bill internal tenants per token, separately decide how to allocate idle capacity, cache residency and shared infrastructure. The H100 experiment does not identify one tenant's true subsidy.

Scope, one last time: the 12–14 pp result is from a controlled multi-tenant vLLM run on one H100 with four synthetic tenants — tight, instrumented, reproducible, but not production billing. The seam percentages are synthetic allocations by construction. The examples demonstrate plausible failure modes, while their percentages describe these constructed scenarios.