Chapter 2 · Book pages 111–124

Systems Foundations for LLMs

Main read ≈ 15 minutes · dotted words have hover/tap definitions · ▸ boxes go deeper · finish with the test (100% to complete)

The big picture

Chapter 1 was the algorithm; this chapter is the machine it runs on. Two subjects: the GPU itself (why it is fast, where its memory lives, when it sits idle) and vLLM, the serving system that squeezes maximum throughput out of it. The recurring idea: performance is about moving bytes, not doing math — the compute units are almost never the bottleneck.

Every API price-per-token you have ever paid traces back to this chapter: how many sequences fit in GPU memory, how full the compute units run, and how much of the hardware sits idle. When part V builds agents that hammer APIs in loops, these economics decide what your agent costs to run.

Stop 1 — Why GPUs at all

A CPU is built for latency: a few cores (8–96) that each finish one thread as fast as possible. A GPU is built for throughput: thousands of simple execution units running in lockstep. Deep learning is dominated by matrix multiplications — O(n³) arithmetic on O(n²) data, embarrassingly parallel — which is exactly the shape GPUs devour. One forward pass of a 70B model needs ~140 TFLOP per token.

GPUs run threads in warps: groups of 32 executing the same instruction on different data (SIMT). If threads inside a warp take different if/else branches, both paths run one after the other — warp divergence. LLM math (matrix multiply, attention, softmax) has uniform control flow, which is why it maps so well.

Go deeper: GPU generations and which one to rent

The lineage: Volta/V100 introduced Tensor Cores (2017); Ampere/A100 added BF16 (2020); Hopper/H100 added FP8 and TMA (2022); Blackwell/B200 doubled again with FP4 and NVLink 5 (2024). Practical picks from the book: 70B+ training → 8×H100/B200 with NVLink; fine-tuning 7–13B → a single A100-80GB with LoRA; budget LoRA → A100-40GB or a 24 GB A10. AMD's MI300X carries the most memory (192 GB) for huge KV caches; Google TPUs are the cloud-only alternative. Book §2.1.1–2.1.3, p. 111–112.

Stop 2 — Inside a GPU

A GPU is an array of Streaming Multiprocessors — an A100 has 108. Each SM is a small independent processor holding CUDA cores (64 scalar units for everyday math), 4 Tensor Cores, a register file, and a slab of on-chip SRAM. Four warp schedulers per SM juggle warps so that while one waits for memory, another computes — latency gets hidden, not eliminated.

The Tensor Core: a hardware unit that does nothing but small matrix multiply-accumulates, extremely fast (book p. 116).

Tensor Cores only pay off when the kernel is compute-bound. At batch size 1 the matrix tiles are tiny, utilization collapses, and you are memory-bound again — the "Tensor Core trap", and the reason inference engines batch requests aggressively (stop 7).

Chapter 1's Flash Attention story is really an SM story: the attention tile fits in the SM's shared SRAM, which is what makes the whole trick work. And FA3/FA4's warp-specialization is about keeping these schedulers and Tensor Cores all busy at once.

Go deeper: SM scaling and precision support

SM counts: V100 80 → A100 108 → H100 132 → B200 148, with SRAM per SM growing 128 KB → 256 KB and L2 growing 6 MB → 128 MB. Tensor Cores accept FP64/TF32/BF16/FP16/INT8, plus FP8 from Hopper — and dimensions should be padded to multiples of 8 (BF16) or 16 (FP8) for full speed. Hopper's WGMMA runs warpgroup-scale tiles (64×256×16) pipelined with TMA data movement. Book §2.1.4–2.1.5, 2.1.9, p. 112–113, 116.

Stop 3 — The memory ladder

GPU performance is almost entirely decided by where your data lives. Each rung of the memory hierarchy is roughly 10× slower and 100–1000× bigger than the one above it. Walk the ladder:

Interactive · The memory ladder (A100 reference)

Step through six storage levels from registers to SSD and watch bandwidth fall off a cliff.

The A100 can compute 312 trillion operations per second but read only 2 TB/s from HBM. That ratio — 156 operations per byte — is the number every optimization in this book is fighting. Do less than 156 FLOPs per byte you load, and the compute units idle.

This ladder explains chapter 1 in one picture: the KV cache lives in HBM (that's why it's the serving bottleneck), Flash Attention's whole trick is staying on the SRAM rung, and CPU offloading (ZeRO) trades down two rungs to fit models that don't belong in HBM at all.

Go deeper: the rungs in numbers

A100 reference: registers ~256 KB/SM at >100 TB/s (1 cycle); shared SRAM 164 KB/SM at ~19 TB/s; L2 40 MB at ~5 TB/s; HBM2e 80 GB at 2 TB/s; CPU DRAM 512 GB+ at ~25 GB/s over PCIe; NVMe SSD terabytes at ~7 GB/s. Register spilling is a performance hazard; L2 hit rates matter for reused weights; PCIe Gen4 is a ~60× step down from HBM per direction. Book §2.1.6, p. 113–114.

Stop 4 — The roofline: memory-bound vs compute-bound

Arithmetic intensity = FLOPs performed per byte fetched from HBM. Plot attainable speed against it and you get the roofline: a rising slope (limited by bandwidth) that flattens at peak compute. The A100's ridge sits at 156 FLOP/byte. Drag the slider:

Interactive · Roofline explorer (A100, BF16)

Move a kernel along the arithmetic-intensity axis and watch it cross from memory-bound to compute-bound at the ridge.

The book's worked example: standard attention at 4K context does ~8.6 GFLOP but drags 138 MB through HBM (four passes over the n² score matrix) → intensity ≈ 62, only 40% of the ridge — memory-bound, GPU 60% idle. Flash Attention cuts traffic to 4 MB → intensity ≈ 2048, 13× past the ridge — compute-bound, full 312 TFLOP/s.

Inside one transformer block the two halves live on opposite sides of the ridge: attention is memory-bound (n² scores), the FFN's big GEMMs are compute-bound. That is why Flash Attention helps attention but not the FFN, and why quantization helps the FFN most.

"Which side of the ridge am I on?" is the most reusable systems question in the book. Prefill vs decode (stop 8), the Tensor Core trap (stop 2), why batch size matters, why speculative decoding wins at batch 1 — all the same question.

Go deeper: the arithmetic, step by step

Attention at n=4096, d=128: FLOPs ≈ 4n²d ≈ 8.6 GFLOP. Standard-implementation traffic: read Q,K (2 MB), write scores (33.5 MB), read for softmax (33.5 MB), write probabilities (33.5 MB), read P and V (34.5 MB), write output (1 MB) ≈ 138 MB → I ≈ 62. Flash: read Q,K,V + write O = 4 MB → I = n/2 = 2048, needing only ~152 GB/s of bandwidth (7.6% of HBM). Book §2.1.7–2.1.8, p. 114–115.

Stop 5 — Connecting GPUs

One GPU is never enough, and moving data between them has its own ladder. Inside a server, NVLink (900 GB/s on H100) connects GPUs directly — with NVSwitch letting all 8 talk at full speed simultaneously. Between servers, InfiniBand (50 GB/s per port) with RDMA lets one GPU write into another's memory across the network, no CPU involved. PCIe (32 GB/s) is the slow lane for CPU↔GPU traffic — never for GPU↔GPU when NVLink exists (28× slower).

Fast inside the box, slower between boxes — the bandwidth gap that dictates parallelism strategy (book p. 117–118).

Parallelism follows bandwidth: tensor parallelism chats every layer → keep it inside a node on NVLink (TP=8 is the DGX standard); pipeline parallelism sends activations point-to-point → may cross nodes; data parallelism and FSDP do an AllReduce of gradients → scale across nodes over InfiniBand.

The book's worked number: syncing gradients for a 70B model across 64 GPUs moves ~275 GB per GPU per step — 0.69 s on a 400 GB/s rail-optimized network, or 41% of a 1-second step. Chapter 11's whole discipline (overlap communication with compute, FSDP, compression) exists because of this number.

Go deeper: primitives, topologies, generations

Collective primitives and their volumes: AllReduce moves 2(N−1)/N of the data (gradient sync), AllGather and ReduceScatter (N−1)/N each (FSDP), point-to-point for pipeline stages, broadcast for weight distribution. NVLink generations: 300 (V100) → 600 (A100) → 900 (H100) → 1800 GB/s (B200). Clusters use fat-tree networks (a 3-level tree of 64-port switches supports 65,536 nodes) and rail-optimized wiring so all 8 NICs per node work in parallel during AllReduce. GPUDirect RDMA: HBM → NIC → network → NIC → HBM, ~1–2 µs latency. Book §2.1.10, p. 116–118.

Stop 6 — The KV cache problem and PagedAttention

At serving time, the KV cache is the biggest memory consumer — for Llama-3 70B it costs ~40 KB per token per GPU, about 1.3 GB for one 4K conversation. Old systems reserved one contiguous slab per user, sized for the maximum possible length. Result: fragmentation — short chats waste their reservation (internal), and freed slabs leave unusable gaps (external). Real utilization: 20–40%.

PagedAttention (the heart of vLLM) steals the fix from operating systems: chop the cache into fixed-size blocks (16 tokens each), keep a per-sequence block table mapping logical positions to scattered physical blocks, allocate on demand. Try it:

Interactive · PagedAttention block allocator

Grow two sequences, share a prefix, and free a finished one — watch blocks scatter with zero wasted gaps.

Waste drops to at most one partly-filled block per sequence (≤15 tokens). Sequences sharing a system prompt share the same physical blocks — copy-on-write, like OS memory. The book's chatbot example: a 1000-token system prompt for 128 users costs 42 GB without sharing, 0.33 GB with it — a 128× saving.

This is a recurring pattern worth collecting: a 50-year-old operating-systems idea (virtual memory) transplanted into ML serving. Preemption-by-swap, copy-on-write, page tables — all reappear here. Every serving stack your future agents run on inherits this design.

Go deeper: block size and the 70B budget

Block size trades table overhead and access locality (bigger blocks) against fragmentation and sharing granularity (smaller); vLLM defaults to 16 tokens, with 32–64 for 100K+ contexts. The book's budget: 70B BF16 on 8×A100-80GB, tensor parallel → 17.5 GB weights per GPU, ~59.5 GB left → ~1.45M tokens of KV capacity → 128 concurrent 4K sequences fit; naive allocation wastes ~50% and fits only 64. Book §2.2.1–2.2.3, 2.2.6, p. 119–121.

Stop 7 — Continuous batching

Old-style "static" batching waits for the whole batch to finish — if one user generates 500 tokens and another 10, the GPU idles for 490 steps on the short one. Continuous batching re-decides the batch every single step: finished sequences leave immediately, waiting ones join instantly. Watch the difference:

Interactive · Static vs continuous batching

Run the same workload both ways and compare wasted GPU slots.

Continuous batching keeps utilization near 100% and delivers 1.5–3× more throughput. It only works because PagedAttention can allocate and free KV blocks mid-flight — the two techniques are a package deal.

Add prefix caching (hash prompt blocks, reuse across requests — 60–80% faster time-to-first-token for chat apps with long system prompts) and you have the reason agent system prompts are affordable at all. Part V's agents lean on exactly this machinery.

Go deeper: speculative decoding and guided decoding in vLLM

vLLM integrates chapter 1's speculative decoding with paging: draft tokens get speculative KV blocks, promoted on acceptance and freed on rejection at O(k) block-table cost, with 3–5 tokens typically accepted per verification. Guided (constrained) decoding runs through pluggable backends — XGrammar (default, pushdown automaton, full grammars) or Outlines (FSM) — masking logits after the forward pass at <1 ms per step and <2% throughput cost; schema compilation (0.5–5 s) is cached per schema. Reminder from the book: structured ≠ correct — valid JSON can still contain hallucinated values. Book §2.2.5, 2.2.12, p. 121, 123–124.

Stop 8 — vLLM end to end

The full system: an OpenAI-compatible API server feeds a scheduler that juggles three queues (waiting, running, swapped) against the block pool. Each request lives two very different lives: prefill — the whole prompt processed in one big compute-bound pass — then decode — one token per step, memory-bound, reading the entire KV cache to produce a single token. When memory runs out, the scheduler preempts victims by swapping their blocks to CPU memory and resumes them later.

Prefill is compute-bound, decode is memory-bound — the roofline (stop 4) applied to serving. This split is why modern datacenters disaggregate the two phases onto different machines (NVIDIA's Dynamo orchestrates exactly that above vLLM and friends).

The payoff table from the book (70B on 4×A100): vLLM vs plain HuggingFace generate — throughput 2,500–4,000 vs 300–600 tokens/s, memory utilization 90–95% vs 50–60%, 200–500 vs 16–32 concurrent sequences, TTFT 100–300 ms vs 500–2000 ms. Same GPU, same model — an order of magnitude from systems engineering alone.

Go deeper: the request lifecycle

Arrival → tokenize, join the waiting queue. Scheduling each step: resume swapped sequences if blocks free up, admit waiting ones if their full prompt fits, budget one new block per running sequence, preempt the lowest priority when over budget. Prefill computes and stores KV for every prompt token at once; decode steps generate one token across the whole running batch; the block manager allocates a fresh block whenever a sequence's last block fills; on EOS the sequence's blocks free instantly for others. Prefix caching hashes each logical block's tokens and skips prefill on hits, evicting by LRU. Book §2.2.8–2.2.11, p. 121–123; Dynamo p. 124.