DSpine for Qwen3.8-27B: Causal Features Through Depth
Explain DSpine's predecessor-feature writes, parallel draft and serial refinement from pinned Qwen3.8 release code. Derive memory costs and serving limits.
AI Blog: LLMs, GPUs and Linux
Guides, comparisons and calculators for LLM inference, GPU memory, NVIDIA hardware and Linux. Browse all technical articles by flozi00 TechHub.
98 technical articles · Browse by topic
Ripple-65B: Embedding Value Residuals and Hybrid Attention
Explain Ripple's embedding-seeded V blend, PeriNorm and local/global attention from code. Derive compact KV capacity and expose the reference cache boundary.
Kimi K3.1 Pro: Sparse Reads, Not a Smaller History
Trace K3.1 Pro's sparse MLA and NoPE indexer from pinned code. Explain top-2048 selection, retained history, approximation error and serving limits.
Nemotron 3.5 Transcribe: From Recorded Speech to Dataset Audits
Trace Nemotron 3.5 Transcribe from audio frames to hybrid MoE: hour-long context math, speaker fusion, memory limits, applications and dataset audits.
Attention Explained Visually
See how attention compares queries with keys and mixes values. A beginner-friendly interactive lesson covering softmax, causal masking and a small example.
The KV Cache Explained Visually
Follow a prompt and new tokens through the KV cache. Learn what keys and values are reused, why memory grows, and what caching does for text generation.
NVFP4 vs INT4 Explained Visually
Compare INT4 and NVFP4 with interactive number lines. Understand rounding error, E2M1 floating point and block scaling without a machine-learning background.
Understand AI Visually: Start Here
Explore interactive diagrams across every technical article: attention, KV caches, model architectures, GPUs, parallelism, serving and Linux.
Granite Speech 5.0 TurboCTC: Pyramidal Conformer Serving
How Granite Speech 5.0 TurboCTC reduces ASR compute with pyramidal Conformer blocks, chunk attention and non-autoregressive CTC decoding.
Nemotron 3.5 Bolt E4B: Block-Parallel Speculation with DFlash
Derive DFlash tensor shapes, parameter and cache bytes, exact speculative verification, NVFP4 limits and the concurrency boundary of Nemotron 3.5 Bolt E4B.
OlmoEarth v1.3: Attention Pooling, Matryoshka Distillation, and Searchlight
Tensor shapes, attention math, Gram distillation, signed-power int8, Zarr layout and production limits in OlmoEarth v1.3.
Nemotron 3.5 Embed: adding images without replacing the text index
Frozen text embeddings, visual-space alignment, 1B distillation, dynamic dimensions and NVFP4 limits in NVIDIA Nemotron 3.5 Embed.
Fibo 2.0: Gist Compression, Dual Text Injection and Clean-Latent Editing
How Fibo 2.0 combines compressed Qwen3-VL gist tokens, direct text injection and clean image latents, with tensor shapes, attention-cost math and runtime limits.
Nemotron 3.5 Super VL: Where Visual Token Compression Happens
Follow paired-frame projection, pixel shuffle and change-based pruning in Nemotron 3.5 Super VL, with tensor shapes, memory math and runtime limits.
Beam 1.0: Local RoPE, Global NoPE and Scalable Softmax
Inspect Beam's 43 local and nine global attention layers, derive SSMax's entropy effect, and separate logical KV payload from serving memory.
Muse Spark Open: Norm-Preserving Gates Are Not a Norm Guarantee
Derive Muse Spark Open's residual-gate coefficients, distinguish three kinds of gates and explain per-expert normalization and safe kernel transformations.
Nemotron 3.5 Bolt: Heterogeneous LatentMoE and the 24 GB Budget
Trace Bolt's heterogeneous experts, 66-byte IQ2 blocks and mixed precision; derive KV and Mamba state without treating 24 GB as peak runtime memory.
Step-5-Preview: Sparse Reads Are Not a Compressed KV Cache
Derive Step-5-Preview's eight-token index summaries, grouped top-k reads and BF16 cache floor, and distinguish serving support from reference-code limits.
Ling-3.1-flash: Grouped Routing and Two Different MLA Caches
Explain Ling-3.1-flash grouped routing, KDA state and MLA weight absorption with tensor shapes, pinned code and explicit cache-memory assumptions.
LFM2.5-24B-A2B: Biased Routing, Short Convolutions and Ten KV Caches
Derive sigmoid MoE routing, gated convolution and GQA cache memory from pinned LFM2.5-24B-A2B weights, configuration and implementation.
Serving Qwen3.8 on MI350P: Kernels, Failures, and Measurements
118 measured runs: GDN state replay, dense QSA, MoE routing, FP4/FP8 tradeoffs, rejected kernels and the limits of throughput, latency and quality claims.
Liquid d1-3B: Branching Decisions Without Decoding
Derive d1-3B tree inference, exact split-attention merging and typed vocabulary readout, with tensor shapes and production boundaries.
Mellum2.1: Sparse Experts, Local Attention and Selective YaRN
Inspect Mellum2.1 tensor shapes, top-8 routing and layer-selective YaRN. Derive its hybrid KV-cache budget and separate released features from MTP announcements.
Clef Omni: Audio, Video, Sparse Experts and Schema Decisions
How Cloudflare Clef Omni combines Qwen3-Omni audio and vision encoders, top-8 MoE routing and schema scoring, with tensor shapes, memory math and deployment limits.
Qwen-Image 2.1: Block-Causal Attention, Prefix KV Cache, and Turbo
Understand Qwen-Image 2.1's block-causal transformer and 8-step Turbo checkpoint. Derive tensor shapes, prefix-cache memory, saved work and production limits.
Inside EmbeddingGemma 2: Soft Tokens, PLE, Hybrid Attention, and MRL
Understand EmbeddingGemma 2's released architecture: media-token fusion, 20 local and four global encoder layers, PLE, 768d pooling, MRL, and production limits.
pplx-embed-v2: 9B documents, 0.6B queries
The architecture and math of 9B document embeddings with 0.6B queries: shared coordinates, MaxSim, measured retrieval quality and serving tradeoffs.
Inside Kolibri 1: Hybrid RNoPE Attention, Sparse MoE, and FP8 Serving
Understand Kolibri 1's released architecture: 40 sliding-window and 10 full-attention blocks, top-6-of-384 MoE routing, cache math and FP8 serving limits.
Inside Cloudflare Clef: Prefill-Only Qwen and a Joint Schema Head
Explore Clef and Clef-flash architecture: prefill-only Qwen backbones, option evidence, joint field attention and production limits.
Inside GLM-5.3-Flash: KDA, Sparse MLA, mHC, and 8-of-288 MoE
A source-checked guide to the released GLM-5.3-Flash checkpoint: its 3:1 KDA/sparse-MLA stack, four-stream mHC residuals, MoE routing, cache shapes, FP8 weights, and serving limits.
Inside Kimi K3: Delta Attention, Depth Attention, and LatentMoE
Read Kimi K3's released architecture: KDA recurrence, global MLA, Attention Residuals, LatentMoE routing and serving limits.
Inside Qwen3.8-Flash-Next: Gated DeltaNet, Sparse Attention, and N-gram Memory
Explore the released Qwen3.8 Flash Next architecture: GDN, sparse attention, MoE routing and offloaded n-gram embeddings.
Agent-Fleet Token Economics Is Cache Economics: Decomposing the 60x Dollar Spread
Reconstruct agent-fleet costs from published token prices, examine an unaudited usage claim and budget for prompt-cache reuse.
BOOST: Concurrent Host and HBM Access for LLM Decode
Compare concurrent host and HBM access with prefetch tiering, and distinguish GH200 throughput results from latency and PCIe extrapolations.
The Cache-Read Price War of September 2026 That Wasn't Three-Sided
Compare published model and cache-read prices from September 21–22, 2026, with per-vendor agent-workload cost calculations.
CXL-SSDs For LLM Prefix Caching: Byte-Addressability Buys Nothing Without Chunk Awareness
Examine CXL-SSD prefix caching, LM-CXD's chunk-aware design, data movement and the paper's TTFT results and deployment limits.
CXMT at 10% of DRAM Revenue: What the Figures Show
Counterpoint's Q2 2026 revenue estimate and CXMT's G5 and LPDDR6 claims, with cost and yield estimates clearly separated from measurements.
DeepSeek-V4.1-Flash KV-Cache Compression: What the Reported 890 Bytes per Token Includes
Understand CED, CSA2, FP4 and bounded replay, and what the paper discloses about DeepSeek V4.1 Flash's reported KV bytes per token.
Disaggregated Quantization: Splitting the Prefill and Decode Price System Across Two Formats
Examine phase-specific quantization, accuracy tests and SSD-streamed prefill, with separate checkpoint costs and runtime measurement limits.
FlashLoop, or: Looping a Transformer Quadruples the Bill and Then Throws Most of It Away
Examine token sparsity, sparse attention and INT4 KV residuals in FlashLoop, with the paper's capacity, speed and accuracy limits.
Gumbel Watermarking Finally Shipped — and the Compliance Asymmetry Nobody Priced
Compare SynthID-Text and vLLM Gumbel watermarking, statistical detection limits and the Article 50 marking duty discussed in the guide.
HBF as the Third KV Tier: 24x Sessions or 5x Worse Latency - the Medium Is Fine, the Placement Policy Decides
Compare high-bandwidth flash KV-cache studies: cold-pool placement, offload traffic, endurance and the latency budget behind capacity gains.
KernelOPT's Agentic Kernel Search: The Verification Cascade Works, The Speedup Fades With Difficulty
Analyze profiling-guided Triton optimization, model-level verification and why vendor-library time limits the study's speedups.
KREX: Shared-GPU Kernel Benchmarking Without Corrupting the Agent's Search
Understand critical-region isolation for shared-GPU benchmarks, how timing bias affects kernel search and the study's measurement limits.
Who Pays for the KV Cache? Metering Rules Are a Pricing Decision in Disguise
Compare raw-token and equal-time billing in a synthetic vLLM study, separating KV capacity, compute time and attribution limits.
Risk-Controlled KV Eviction: The Reliability Contract and Full-KV Fallback
Understand retention calibration, finite-sample guarantees and full-KV fallback in risk-controlled eviction, with reproducible examples.
KITE: Scale the Model, Freeze the KV Bill — the Prefill Invariant arXiv:2609.27294 Actually Proves (and the Three Things It Does Not)
Understand KITE's prefill KV invariant, two-tower architecture and the limits of inferring capability or decode costs from training loss.
KV-Cache Tensor Decomposition: Full-Rank Head and Layer Modes in Two Models
Examine head and layer rank, Tucker decomposition and reconstruction errors in two studied models, with an illustrative simulation.
KVSET: LRU Stack Distance for LLM Prefix-Cache Sizing
Use LRU stack distance to derive a prefix-cache hit curve from one trace. Includes a worked simulation and workload limits.
Leaderboard Margins vs Hidden Model Selection: Which Gains Survive k Secret Variants?
Explore how hidden model selection affects benchmark margins, with a Gaussian model, correlation assumptions and a simulation.
Greedy Decoding Is Not Precision-Invariant: BF16-vs-FP16 Flips, Margin Gating, and the Kernel-Order Axis
Understand logit margins and reduction order in greedy decoding, with numerical examples and the limits of reported mitigations.
LLM Prefix Caching: Where LRU Holds Up and Where It Falls Short
Examine LRU on production prefix-cache traces, compute-weighted savings and the workload and capacity limits, with a runnable simulation.
PTQ Configuration Pricing: Predicting the Quantization Price Before You Build the Model
Understand curvature-weighted error pricing for post-training quantization, calibration traces and the limits of the quadratic approximation.
Quantization Can Change Retrieval Results While Classification Accuracy Holds
Examine retrieval-ranking changes after 4-bit quantization, the score-gap bound and proposed mitigations without equating changed ranks with lost relevance.
Snapdragon X2 on Linux: The 80-TOPS NPU Is Not the Story — 228 GB/s Is
Examine Snapdragon X2 Linux support and memory bandwidth, with illustrative local LLM decode estimates separated from measured performance.
Uncheatable Eval: Scoring LLMs by How Few Bits They Need
Convert compression rate and bits per byte, then examine the benchmark's scaling results and limits of contamination control.
KV Cache Memory Growth Explorer
Watch one agent's KV cache fill a node's free HBM as context grows — across GQA, MLA and hybrid attention models, with a live console fill bar per model.
TypeSafe Jev: What the Public Evidence Supports
A careful reading of TypeSafe's Jev launch, workflow evaluations, calibration claim, and relevant prior art.
GLM-5.3 MTP on B300: Acceptance Math and NCCL NVLS
A worked example of MTP acceptance-rate math for GLM-5.3-style serving, plus documented changes to NVLS failure behavior across NCCL versions.
Your Agent Fleet Is a Storage Array: The KV-Cache Tiering Math Behind Agentic Inference
Calculate KV bytes moved per agent turn, fleet bandwidth and working-set capacity, separating PCIe transfers from HBM access.
EU AI Act Article 50: Marking and Its Technical Limits
Article 50(2) requires providers of generative AI systems to mark synthetic outputs, subject to exceptions and technical feasibility. What the 2026 transition means, and what watermark research does and does not prove.
EU AI Act after the Digital Omnibus: What Moved
Regulation (EU) 2026/1744 postponed two high-risk application dates. The prohibited-practice, GPAI and Article 50 rules have separate dates. A source-based guide to the change.
EU AI Act and GDPR for Self-Hosted LLMs
Which EU AI Act and GDPR obligations can apply to a self-hosted LLM: system roles, high-risk uses, transparency, open-source GPAI and personal-data processing, checked against primary sources.
Random Attention vs Score-Based KV Eviction on Reasoning Traces
Examine random eviction with a pinned prompt on reasoning traces, compare score-based baselines and calculate KV traffic per token.
Self-hosting DeepSeek V4 Pro: the actual math
Calculate MoE weight storage, KV-cache growth and decode ceilings for DeepSeek V4 Pro, with a conditional API cost comparison.
GPU Buying Guide for LLM Inference (2026)
How to read GPU specs for LLM serving: model fit, peak bandwidth, interconnect, purchase quotes, used hardware and workload choices.
HBM Capacity and KV-Cache Economics
How context length consumes GPU memory, why HBM uses more wafer input per bit than DDR5, and what NVIDIA and TrendForce have actually said about Rubin and Rubin Ultra capacity.
Germany's KI-MIG: Who Enforces the AI Act?
Germany's KI-MIG names the Bundesnetzagentur as the default AI Act market-surveillance authority, with statutory sectoral exceptions. A source-based map of powers, registration and application dates.
The KV Cache: Bit-Exact Memory Math, GQA vs MQA vs MLA, and PagedAttention
Calculate KV-cache memory per token and request. Compare GQA, MQA and MLA, understand PagedAttention, and work through GPU capacity examples.
KV-Cache Glossary: every term in the serving stack
Definitions and formulas for KV bytes per token, PagedAttention, prefix caching, eviction, TTFT, TPOT and memory tiering.
llama.cpp / Ollama vs vLLM: Which One, When, and Why
Choose a local LLM stack: compare llama.cpp, Ollama and vLLM, GGUF support, concurrency and a worked 14B model VRAM budget on a 24 GB GPU.
LoRA and QLoRA Fine-Tuning: The Actual Memory Math
Mixed-precision AdamW's 16-byte model-state budget, LoRA's r(d_in+d_out) adapter math, QLoRA's NF4 storage and the limits of activation-memory estimates.
NVIDIA Groq 3 LPX: The SRAM Decode Engine and the Arithmetic NVIDIA Won't Do
Analyze Groq 3 LPX's SRAM capacity and bandwidth, compare hardware ratios and distinguish token latency from aggregate throughput.
NVIDIA Agrees to Buy Hugging Face: The Dependency Math Behind the $12.93B Headline
Map Hugging Face dependencies in vLLM, SGLang and TensorRT-LLM, and examine the supply-chain questions raised by the announced acquisition.
LLM Quantization: Bit Layouts, Block Scales, and the VRAM Math
Compare FP8, NVFP4, MXFP4, GPTQ, AWQ and GGUF quantization. Worked bit-layout and block-scale examples explain how much VRAM each format needs.
RAG Retrieval Math: Embedding Memory, Vector Index Size, and the Latency Budget Nobody Calculates
Calculate embedding weights, vector storage and retrieval latency for self-hosted RAG, with flat search, HNSW and reranker examples.
The 2026 Serving-Engine Churn Tax: vLLM 0.29, SGLang 0.5.19, and TensorRT-LLM Losing TensorRT
Review breaking vLLM, SGLang and TensorRT-LLM releases from September 2026, with upgrade costs, version pinning and rollback planning.
UNISON and the Paused Fleet: Why Agent KV Traffic Breaks Every Cache Policy You Know
Analyze paused-agent working sets and UNISON's SPEAR and TIDE mechanisms, with KV traffic arithmetic and a software comparison.
vLLM vs SGLang: Architecture, Overhead, and When to Pick Which
Compare vLLM and SGLang for LLM serving: PagedAttention, RadixAttention, scheduling, quantization and a decision table for your workload.
AMD's MI455X 34× Throughput Claim: What the Disclosure Establishes
Examine AMD's MI455X throughput claim against disclosed memory and bandwidth ratios, and see which benchmark details are still missing.
AMD MI300X/MI350/MI400: More Memory, But Does the Math Win? — A Critical Silicon View of AMD Instinct vs NVIDIA
Compare AMD MI300X, MI350 and MI400 with NVIDIA GPUs: sourced memory and bandwidth specs, quantization support and limits of peak performance figures.
MLPerf Inference 2026: What a Result Says About Hardware and Software
Read MLPerf Inference v6.0 and v6.1 results with software improvements, cluster size and preview submissions kept in context.
Scale-Up vs Scale-Out: Network Bandwidth and Collective Math
Compare NVLink domains and inter-rack networks with a worked ring all-reduce example and explicit bandwidth assumptions.
How LLMs Actually Work — An Animated Walkthrough
From token to probability distribution: embeddings, attention, training and sampling — explained without the equation jump, with animation.
LLM Inference Calculator & Break-Even Analyzer
Estimate LLM VRAM, KV-cache memory and theoretical token throughput for Qwen, DeepSeek and GLM. Compare GPUs, precision and batch sizes.
LLM Inference Economics: Costs & Cloud Break-Even
Build vs. buy for LLM inference: cost per token, amortization and break-even for GPUs from RTX PRO 6000 to B300, side by side.
Parakeet TDT vs Encoder–Decoder ASR: Production Trade-offs
Where Parakeet (FastConformer + TDT) fits production ASR: streaming, long audio, timestamps and N-gram LM shallow fusion.
Expert Parallelism: The Router's Dilemma
Expert Parallelism for MoE models: routing tokens instead of matrices, all-to-all communication, load imbalance and real implementations.
Pipeline Parallelism: Pumping Data Upstream
Pipeline Parallelism explained: split layers vertically across GPUs, schedule with 1F1B, and keep the pipeline bubble under control.
Tensor Parallelism: Slicing the Silicon
Tensor Parallelism explained: how weight matrices shard across GPUs, what all-reduce communication it costs, and when TP stops scaling.
Selecting the Right GPU for Qwen3 Inference
Compare RTX PRO 6000, H200 and DGX Station for Qwen3 inference. Worked examples explain model fit, GPU bandwidth and theoretical decode limits.
LLM Inference Math: From Theory to Hardware
Understand LLM inference bottlenecks: calculate bytes per token, GPU bandwidth and theoretical decode speed with a worked Qwen3-VL-32B example.
LLM VRAM Requirements: A Mathematical Deep Dive
How much VRAM does an LLM need? Calculate weights, quantization and KV-cache memory step by step, with a Qwen3-VL-32B example and a VRAM calculator.
Qwen3-Next-80B-A3B: Architecture and Serving Limits
The model card's hybrid Gated DeltaNet and attention layout, sparse experts, memory requirements, and deployment guidance.
NVIDIA GPU Containers on Ubuntu 24.04
Install a host driver, Docker Engine and the NVIDIA Container Toolkit using their current official instructions, then verify GPU access.
NVIDIA Error 802 on Ubuntu: Diagnose Before Trying nokaslr
What a forum report about nokaslr actually establishes, and how to distinguish CUDA initialization and Fabric Manager failures.
NVIDIA B200 vs GB200: What MLPerf Training Actually Shows
B200 vs GB200 explained: compare system design, NVLink domains and published MLPerf training results, with the limits of performance comparisons.
NVIDIA NVLink: Generations, Domains and Verification
Compare NVLink bandwidth by generation, understand NVSwitch and HGX vs NVL72 domains, and check GPU topology on Linux with nvidia-smi.