Who Pays for the KV Cache? Metering Rules Are a Pricing Decision in Disguise

Every FinOps conversation about self-hosted inference eventually collides with a question the public clouds answer with a price sheet and the private ones answer with a fight: when four tenants share one GPU and one KV cache, who pays? The API providers already answered it in public — cache reads at a tenth of input, cache writes at a premium, the whole thing settled by fiat in the rate card we tracked in the cache-read price war. On a Kubernetes cluster running your own vLLM, nobody hands you a metering policy. You pick one, and the pick silently re-prices every tenant.

That is the entire subject of arXiv:2609.24991, "Who Pays for the KV Cache? Attributing Shared AI Inference Spend Across Kubernetes and LLM Provider Bills" by Timothy Urista (submitted September 21, 2026)¹. The paper has two halves that are usually kept apart: a tool, unalloc, that joins OpenCost, LiteLLM, OpenAI and Anthropic cost data into one decimal-exact ledger and reports the share of spend with no owner; and a measurement, on an H100 running vLLM, of how much the metering rule alone — not the workload, not the hardware, just the billing key — moves money between tenants. We verified both halves at the primaries; this guide decomposes the second, because it is the one that generalizes. If you run agentic workloads with heavy prefix reuse, it is about your bill.

The headline, verified

On a DigitalOcean H100 80GB droplet (a 29-minute, approximately $2.15 run, fully captured in the repo), vLLM 0.29.0 served Qwen2.5-7B-Instruct in bf16 with an 8,192-token context and prefix caching; vLLM sized its KV pool at 995,296 tokens². Four tenants share the pod. Across configured loads of 2 to 16 requests per second — 3.7 to 26.9 completed requests per second, because the configured rate counts session-initial arrivals only — the two meters disagree at every load²:

The abstract rounds this to "12–14 percentage points at every load tested"; the table's exact gaps are 11.7–13.6, a rounding we confirmed ourselves from Table 3 of the paper². The direction never flips and the gap widens with load. Meanwhile GPU utilization reads 97–99% across all four loads and power draw tracks load — the box is genuinely busy at every setting, which is what makes the disagreement a metering problem and not an idle-capacity artifact².

Also verified, and worth stating plainly: all 6,241 requests completed without error, and the served model stores about 57 KB of KV per token under the paper's own accounting².

Why a token meter misprices retrieval and agentic tenants

The mechanism is the one our agent-fleet economics piece kept bumping into from the provider side: cache-hit-heavy tenants consume GPU-seconds — their KV sits resident in HBM, occupying paged blocks, holding memory hostage for the next turn — but they bill almost no fresh tokens. A token meter prices throughput; the scarce resource in a shared prefix-caching server is residency. Those are different commodities, and any per-token meter systematically under-prices the commodity the cache-heavy tenant consumes most.

Run the units. One paused agent session holding 32,768 tokens of resident KV on this exact configuration:

# H100 run from arXiv:2609.24991 Sec. 9: 57 KB KV/token, 995,296-token pool, 80 GiB HBM
# illustrative on-demand H100 hour: $2.50, held constant for the whole guide
GiB   = 1024**2               # in KB
kv_per_tok_kb = 57            # paper-measured, Qwen2.5-7B bf16
hbm_gib       = 80
hourly        = 2.50
 
paused_tokens = 32_768        # one paused agent session's resident KV
held_gib = paused_tokens * kv_per_tok_kb / GiB
residency_per_hour = hourly * held_gib / hbm_gib
print(round(held_gib, 2))              # 1.78
print(round(residency_per_hour, 4))     # 0.0557
print(round(paused_tokens / 995_296, 3))          # 0.033

A single paused session holds 1.78 GiB — about 3.3% of the entire pool — and its pro-rata share of the GPU is roughly $0.06 per hour it sits there³. Now the meter question as a month:

month_hours = 24 * 30
held_gib = 32_768 * 57 / (1024**2)
resid_month = 2.50 * held_gib / 80 * month_hours   # holding one paused session all month
 
# the same session re-sends its whole context 24x/day: all cache reads
read_tokens_month = 32_768 * 24 * 30
cache_read_bill   = read_tokens_month / 1e6 * 2 * 0.1   # 0.1x of a $2/1M input price
print(round(resid_month, 2))     # 40.08
print(round(cache_read_bill, 2)) # 4.72
print(round(resid_month / cache_read_bill, 1))  # 8.5

Held resident for a month, the session's pro-rata GPU cost is about $40. The token-style bill for its 23.6M cache-read tokens, at the aggressive 0.1x read discount providers advertise, is under $5 — an 8.5x mispricing of residency as tokens. That gap is measured structure, not modeled pessimism: the author's companion write-up reports, for the paper's simulator at 3 req/s, a 75% overall cache hit rate with the agents tenant at 92.6% prompt-token hits and search at 33.7%⁴. (Those simulator-tenant numbers are from the synthetic case study, not the H100 run — the label matters, and we keep it.)

So the paper's 11.7–13.6 points are not noise; they are the measurable price of treating bandwidth-seconds as tokens. The arithmetic above is the same physics and is why the direction of the gap cannot flip: the token meter hands the retrieval tenant prompt bytes it mostly re-reads from cache, while the time meter hands it wall-clock its batches share with everyone.

Neither meter is ground truth — and the paper says so

The single most respectable sentence in the paper is that neither meter is a ground truth, with the results positioned against exact Shapley-based energy attribution work¹⁵. Equal time-share is itself a heuristic — it charges a request waiting on prefill the same as one decoding. List-price weighting (cached input at 0.1x, output at 4x) is no better: in the simulator it moves the agents tenant twelve points away from measured step time². The token meter and the time meter bracket the truth for a shared server, and the paper's recommendation is the right one: decide the meter explicitly, publish the shares, and stop pretending the ledger's total authority extends to its split. This is the on-prem mirror of the provider-side read-pricing fight: someone pays for cache residency; the only question on your cluster is whether you chose who, or let the join key choose.

The Kubernetes seams: excellent findings, synthetic clothes

The other half of the paper — where attribution breaks between Kubernetes allocations, gateway logs, and provider bills — is legitimately good failure-mode engineering, and the paper itself is scrupulous about its scope in a way the secondhand summaries are not. These are probes of a constructed multi-pod scenario: one month of synthetic OpenCost allocations, not observed billing data¹:

• Labels set only on LeaderWorkerSet leader pods leave 66% of that deployment's GPU bill unowned (65.9% in the detailed run)².
• The natural fallback key drops the headline unallocated share to 4% (4.4%) — by routing $23,597, i.e. 61% of the bill, into a bucket called vllm, the Helm chart's app.kubernetes.io/name, which collides with the LeaderWorkerSet canonical key on name².
• Enabling every source double-counts all gateway spend¹.
• Reading one page of a billing API reports a quarter of spend¹.

Read the second bullet twice, because it is the real contribution: the fallback does not fix attribution, it converts unknown spend into wrongly-owned spend while making the headline metric look better. A tool that only reports an unallocated percentage cannot distinguish that scenario from a correct one; the paper's fix is reporting the fallback-attributed amount separately and making collision resolution order-independent².

We mark all four seam findings synthetic-allocation-scenario, paper-verified — real failure modes, demonstrated on constructed data, with raw outputs in the repo⁶. The communication finding generalizes the same caution: GPU time in NCCL collectives totals $12,059 of the synthetic month, and a per-token showback of it moves $3,737 from the tensor-parallel tenant to the pipeline-parallel one².

The ecosystem is moving the same direction

This is not a lone-wolf problem. OpenCost 1.121.0 (release tagged July 20, 2026; announced August 5, 2026 on the CNCF blog) shipped "AI Inference Costs v1" — first-of-a-kind Kubernetes inference cost tracking built on vLLM's vllm:prompt_tokens_total and vllm:generation_tokens_total metrics, validated on a 109-GPU, 30-model proof-of-concept cluster, exposing llm_total_hourly_cost and llm_cost_per_million_tokens through Prometheus and the REST API⁷⁸. The design distinction it makes is exactly this paper's thesis dressed as plumbing: allocation-based cost per million tokens is the full hosting cost — weights, GPU, and a share of gateway and KV-cache storage — while usage-based cost counts only infrastructure consumed during active inference, credits KV-cache-hit savings, and attributes cost to the tokens actually processed⁷. The gap between the two is the cost of keeping the model warm and ready; the CNCF post's own example is a usage-based $1.00/M versus an allocation-based $4.00/M — a 25% utilization ratio⁷. That gap is the same quantity the paper's meters bracket tenant-by-tenant, and the KV-cache-hit credit is where the metering decision lives: it is a pricing policy — someone decided cache-resident reads are cheaper — exactly the decision arXiv:2609.24991 says must be made explicitly rather than inherited from a metric name. Workload and team attribution remain pending on the OpenCost side⁷⁹, so the open question — which meter feeds those lines — is being decided per-operator, mostly by accident. unalloc itself is on PyPI at 0.2.3 — the release the paper's archive and fixtures are pinned to — joining the four major sources, with OpenCost's allocations as one input among several¹⁰¹.

The practical checklist, for anyone running a shared inference server today:

1. Decide the meter in writing — token share, time-share, or a hybrid — and put the decision in the showback itself, not the wiki.
2. Report fallback-attributed spend separately from unallocated spend. Never let a Helm chart name own a bill.
3. Publish both meters when tenants dispute. The 12-point spread is the size of the argument you will otherwise have.
4. If you bill internal tenants per token, know that you are subsidizing cache-resident agentic tenants by roughly the residency arithmetic above — decide whether that subsidy is the product.

Scope, one last time: the 12–14 pp result is from a controlled multi-tenant vLLM run on one H100 with four synthetic tenants — tight, instrumented, reproducible, but not production billing. The seam percentages are synthetic allocations by construction. The failure modes are real; the numbers are demonstrations. In this genre that is not a weakness — it is the only honest way to publish.