Preface

When a model becomes a system

Training gives a model its capabilities. Inference is where those capabilities meet a user.

Every generated token, classified image, synthesized frame, or agent action is an inference event. It arrives with a deadline, consumes memory and compute, and competes with other work. A model can be excellent in an evaluation and still make a poor product if its first response is slow, its stream stalls, or its cost rises faster than demand.

This is why inference matters. Training happens periodically; inference happens every time the product is used. Better inference does more than lower a bill. It makes longer contexts practical, admits more concurrent users, shortens the feedback loop of an interactive system, and allows capable models to run on a wider range of hardware. The 2025 Stanford AI Index reported that the cost of querying a system at roughly GPT-3.5 capability fell more than 280-fold between November 2022 and October 2024. Gains of that scale change which applications are possible—and cause demand to grow in return.

The difficult part is that a model does not run alone. A production request passes through an API, a tokenizer or media processor, queues, a scheduler, accelerator kernels, memory managers, and an output stream. At cluster scale it also crosses routers, caches, interconnects, replicas, and failure boundaries. The model defines the computation. The serving system decides when and where it runs, which state survives, and whether the result reaches the user on time.

That serving system is the subject of this book.

How inference became a systems discipline

The field did not begin with large language models. Early production serving systems concentrated on loading model versions safely, exposing stable APIs, and batching independent predictions. TensorFlow Serving, open-sourced in 2016 and described in a 2017 systems paper, made model lifecycle management and high-performance serving first-class concerns.

The workload changed as the models changed. The Transformer, introduced in 2017, made attention the foundation of a highly parallel model architecture. Large generative models built on that architecture do not simply run once per request. They first process an input and then repeatedly execute the model to produce new tokens, retaining attention state between steps. Requests have different input lengths, finish at different times, and grow their memory footprint while running. A static batch is a poor fit for that shape of work.

A short history of generative-model serving.

flowchart LR
    A["2016–2017: model serving and Transformers"] --> B["2022–2023: continuous scheduling and paged KV memory"]
    B --> C["2024: structured programs and prefix reuse"]

In 2022, Orca showed that a generative serving system could schedule at the granularity of a model iteration instead of waiting for an entire request batch to finish. That idea—now commonly called continuous batching—lets completed requests leave and new requests join between decoding steps.

In 2023, the vLLM PagedAttention paper identified the key-value cache as a central memory-management problem. Its solution borrowed the idea of paging from operating systems: request state could occupy non-contiguous blocks and be shared safely, reducing waste and making larger batches possible.

In 2024, SGLang expanded the unit of optimization beyond a single prompt. Real applications contain repeated prefixes, branching calls, tool interactions, constrained outputs, and parallel generations. Its runtime used radix-organized prefix state and structured-generation optimizations to execute those programs as a whole.

These milestones changed the central question. Inference performance was no longer only about making one model invocation faster. It was about coordinating many evolving requests and their state across a finite machine.

Why vLLM and SGLang

Papers explain a design by isolating its contribution. Production repositories show what happens when that contribution must coexist with everything else: API compatibility, model diversity, numerical formats, distributed execution, hardware backends, observability, failures, and a changing user workload.

vLLM and SGLang are valuable because their code exposes two actively developed answers to the same serving problem. Their schedulers, cache managers, model runners, distributed layers, and tests turn abstract trade-offs into concrete decisions. Studying both helps separate a durable principle from one project’s current implementation. Where they converge, there is usually a shared systems constraint. Where they differ, there is usually a trade-off worth understanding.

This edition studies reproducible snapshots: vLLM at commit 5cecfc0 and SGLang at commit e161bd1, with a manuscript snapshot date of August 23, 2026. The code will continue to change. The book therefore organizes the implementation details around more durable ideas: scheduling, state ownership, data movement, service contracts, and evidence.

Chapter 0 begins with one request on one model worker. From there, the book opens the engine, crosses accelerator and machine boundaries, and finally confronts the operational and economic decisions of a production service. New readers should start with Chapter 0 and continue in order. Experienced practitioners can use the table of contents to enter at the problem they are solving and consult the appendices only when they need a worksheet, reference, or debugging procedure.

By the end, you should be able to take an unfamiliar inference system, draw its critical path, locate the state and queues that govern it, predict where it will saturate, and design a measurement that can prove your diagnosis wrong. That is the craft this book aims to teach.

Part I — The Inference Problem

Before anything can be made fast, it must be made legible. Chapter 0 follows one request through one model worker to build concrete intuition; the remaining four chapters fix the vocabulary the rest of the book argues with: what work actually arrives, what a model creates and must remember, where computation and state can physically live, and what the service promised its users.

Chapters 0–4

0. Your First Inference Request

You have a trained model and enough GPU memory to load it. A GPU is an accelerator designed to perform many numerical operations in parallel. What happens when you send the model a prompt—the text that asks it to produce a response?

Before routing, scheduling theory, or cluster management, there is one request and one model replica—an independently serving copy of the model. This chapter follows that request from the moment text leaves a user’s keyboard to the moment an answer finishes streaming back. The replica may span several GPUs, but we treat it as one logical worker and postpone communication between devices. Every step will reappear, with complications, in later chapters—but here it is one loop, short enough to hold in your head.

The model on the wire

Assume the fictional model used throughout this book’s exercises: “Atlas,” a dense, decoder-only Transformer with approximately 70 billion parameters. A parameter is one learned number in the model. Dense means every generated token uses the same set of parameters; decoder-only means the model produces a continuation one token at a time.

Atlas stores each parameter in BF16, short for Brain Floating Point 16. BF16 is a 16-bit number format commonly used for neural-network weights; each value occupies two bytes instead of the four bytes used by 32-bit floating point. Chapter 10 explains the accuracy and performance trade-offs of reduced precision. For now, its important property is simply its size.

The model has 80 Transformer layers—repeated processing blocks containing attention and feed-forward calculations, the latter being learned matrix transformations applied to each token position. We will introduce the shape of its attention state only when we calculate the request’s memory use later in the chapter. The Atlas constants are collected in the reference card near the end.

One request through one model worker: the complete path.

flowchart LR
    A["Text in"] --> B["Tokenizer"]
    B --> C["Prefill"]
    C --> D["KV cache"]
    D --> E["Decode step"]
    E --> F["Sample"]
    F --> G["Detokenize"]
    G --> H["Text out"]
    F --> D

The arrow from sampling back into the KV cache represents autoregressive generation: the model generates one token, appends it to the sequence, and uses the entire sequence to choose the next token. The loop runs until a stop condition fires. Everything before prefill is string manipulation; everything after sampling is string manipulation. The GPU work lives in the middle, and that middle is where time goes.

The weight footprint is direct arithmetic:

70 billion parameters × 2 bytes = 140 GB

That 140 GB must sit in GPU memory before the model can answer anything. It does not fit on a single 80 GB GPU, so a practical replica divides the weights across multiple devices. For now, assume that replica is loaded and ready and follow the request as if the devices formed one worker. Chapter 4 explains how the devices are connected; Chapter 13 explains how the model is divided among them.

Text to tokens

A model does not read text. It reads integers.

A tokenizer converts text into the integers a model accepts. It usually splits a string into subwords: pieces that may be a whole common word, part of a rare word, punctuation, or whitespace. Each piece maps to an integer ID from a fixed vocabulary. The mapping is deterministic for a given tokenizer version: the same string always produces the same IDs.

"The quick brown fox" → [464, 4996, 8516, 3143]

Four tokens. The vocabulary typically contains 32,000 to 128,000 entries, covering common words, word fragments, punctuation, and whitespace. Rare words are split into several tokens; common words are single tokens. The cost of tokenization is measured in microseconds per token — negligible next to what comes after.

A chat template is a formatting rule that places role markers, system instructions, and separators around a conversation before tokenization. The result is a sequence of integer IDs, typically hundreds to tens of thousands of them, ready for the model.

What the model is, physically

The model is a stack of Transformer layers. Its parameters are also called weights. Most sit in matrices—large two-dimensional arrays of numbers—that transform one array of numbers into another. The 80 layers are applied in sequence: the output of layer 0 feeds into layer 1, layer 1 into layer 2, and so on through layer 79.

Before the first layer, an embedding table converts each token ID into a vector, a fixed-length array of numbers the model can process. After the last layer, an output projection converts the final vector into one score for every token in the vocabulary.

All of these weights sit across the replica’s GPU memory, occupying the 140 GB computed above. They do not change during inference. The model reads them, repeatedly, every time it processes a token.

Prefill: processing the prompt

The first phase of inference is prefill: processing the entire input before generating any output. All input tokens pass through every layer together. This is a large matrix computation in which the model can apply one weight read to many token positions.

For a 1,000-token prompt, prefill performs roughly 1,000 positions’ worth of arithmetic while reading the 140 GB of weights once. Its arithmetic intensity—the amount of calculation performed per byte moved from memory—is high, so the GPU’s compute units stay busy.

Using the service-time model that recurs throughout this book:

prefill_ms(tokens) = 20 + 0.035 × tokens

The 20 ms is fixed overhead: launching kernels (small GPU programs), allocating working memory, and performing initial data movement. The 0.035 ms per token is the incremental cost once the pipeline is running. For a 1,000-token prompt:

prefill_ms(1000) = 20 + 0.035 × 1000 = 20 + 35 = 55 ms

Fifty-five milliseconds from receiving the prompt to completing the first phase. This is the dominant component of time to first token (TTFT) — the delay the user perceives before the answer starts streaming.

What prefill creates

Prefill does not just produce output. It creates persistent state.

Inside each Transformer layer, the attention operation lets a token retrieve relevant information from earlier tokens. The current token produces a query vector. Every earlier position has a key vector used to measure a match with that query and a value vector containing the information to retrieve. Several attention heads perform this matching in parallel from different learned perspectives.

The keys and values must remain in GPU memory because every future decode step will read them. Keeping them avoids recomputing the entire prompt for every new token.

This persistent state is the KV cache (key-value cache). Atlas stores eight key-value heads per layer, and each key or value contains 128 BF16 numbers. Its size per token across all layers is therefore:

2 (keys and values) × 80 layers × 8 KV heads × 128 dimensions × 2 bytes
= 327,680 bytes
≈ 320 KiB per token

For the 1,000-token prompt, the total KV cache created during prefill is:

1,000 tokens × 320 KiB = 320,000 KiB ≈ 312 MiB

Here GB means a decimal billion bytes. KiB, MiB, and GiB are binary memory units: each is 1,024 of the preceding unit. Three hundred and twelve MiB of state, created in 55 milliseconds, that must remain resident in GPU memory for the entire duration of the request. This state will grow by 320 KiB with every new token the model generates.

Decode: generating the answer

After prefill, the model enters the decode phase: the repeated loop that generates one new token at a time. Each decode step:

1. Reads the model weights (140 GB).
2. Reads all accumulated KV cache entries.
3. Computes one new position’s worth of arithmetic.
4. Writes one new KV entry per layer.
5. Produces a vector of logits — one score per vocabulary entry.

The critical difference from prefill is that the model now reads 140 GB of weights to compute only one new position per active request. The ratio of data moved to useful computation is poor. Decode is memory-bandwidth-bound: its speed is limited by memory bandwidth, the number of bytes the GPU can move per second, rather than by how quickly it can perform arithmetic.

Each decode step takes approximately 45 ms for a small group of simultaneous requests in the Atlas cost model. Most of that time is spent streaming weights from GPU memory through the compute units.

Sampling: from scores to a token

The logits produced by the final layer are raw scores, one per vocabulary entry. They are not probabilities yet. To select the next token:

1. Temperature rescales the logits. Lower values make high-scoring tokens more dominant; higher values make alternatives more likely.
2. Softmax converts the scores into probabilities that sum to one.
3. Optional filters reduce the choices: top-k keeps the k most probable tokens, while top-p keeps the smallest set whose cumulative probability reaches a chosen threshold.
4. The server samples from the remaining probabilities. Greedy decoding instead always chooses the highest-scoring token.

The result is one integer: the ID of the next token. This step is small compared with running the model, but it carries state. Sampling uses a pseudo-random number generator: a deterministic sequence controlled by a seed and its current position. Preserving that state is necessary when a system promises repeatable output.

Detokenization: back to text

The selected token ID is converted back to text by the tokenizer’s reverse mapping. The text fragment is sent to the user immediately — this is streaming. The user sees partial words assemble into sentences while the model continues generating.

The loop

The decode loop repeats: read weights, read KV cache, compute one position, write one KV entry, sample, detokenize, stream. Each iteration takes roughly 45 ms and produces one token. A 200-token response takes about 200 steps, roughly 9 seconds of decode time.

The loop ends when one of these conditions is met:

• The model emits a special end-of-sequence token.
• The response reaches a caller-specified maximum length.
• The user closes the connection.

The total time for a 1,000-token prompt with a 200-token response is approximately:

prefill:   55 ms
decode:   200 × 45 ms = 9,000 ms
total:    ~9.1 seconds

The user experiences 55 ms of waiting, then roughly 9 seconds of streaming text at about 22 tokens per second. This is one request, served alone, on hardware with nothing else to do.

Following the bytes

The request owns memory at three different lifetimes.

flowchart TB
    A["Server lifetime"] --> B["Model weights"]
    C["Request lifetime"] --> D["Growing KV cache"]
    E["Step lifetime"] --> F["Activations"]
    E --> G["Logits"]
    B --> H["Read every prefill and decode pass"]
    D --> I["Released when the request ends"]
    F --> J["Reused after each step"]
    G --> J

Activations are the temporary intermediate vectors produced while executing a layer. They can be reused after the step. Logits are the raw vocabulary scores just introduced. A summary of where memory goes during this request:

The weights dominate the memory budget. The KV cache is the only object that grows during the request. Activations are temporary workspace. On hardware with 80 GB of device memory per accelerator, even a single request’s KV cache is a small fraction of capacity — but this changes fast.

Why one request is misleading

The single-request story above is clean. The GPU does useful work, the user gets an answer, and memory is comfortable. Now consider what happens when load increases.

Ten concurrent requests

Ten users send prompts at approximately the same time, so ten requests are concurrent—in progress together. Each has a 1,000-token input. The model weights are still 140 GB: they are read, not copied, so ten requests do not need ten copies. But the KV cache is per-request:

10 requests × 312 MiB = 3.12 GiB after prefill

After each request generates 200 tokens of output:

10 × 375 MiB ≈ 3.66 GiB of KV state

Still manageable on an 80 GB device. But a benefit appears: batching, or processing a group of requests in one model step. The engine reads the 140 GB of weights once and applies them to all ten sequences. The same weight traffic that served one request now serves ten. Each step takes longer—there is more KV state to read and more arithmetic to perform—but not ten times longer.

This is the fundamental efficiency gain of batched inference: the cost of reading the weights is shared across sequences.

One hundred concurrent requests

Push further. One hundred concurrent requests with 1,000-token prompts:

100 × 312 MiB = 30.5 GiB after prefill

On an 80 GB device holding part of the 140 GB model—or holding a compressed version that uses fewer bits per weight—30 GiB of KV state is a significant fraction of the remaining memory. After each request generates 200 tokens:

100 × 375 MiB = 36.6 GiB

And this assumes 1,000-token prompts. A 4,000-token prompt produces 1.22 GiB of KV state per request. One hundred such requests need 122 GiB — more than the entire device. The model cannot even hold them all in memory simultaneously.

The tension

More requests sharing a decode step means better utilization—a larger fraction of the GPU is doing useful work. But more requests also means more KV cache memory. The engine faces a direct trade-off:

• Admit more requests: better throughput, higher GPU utilization, but more memory pressure. Eventually the engine must evict cached state or preempt a request—pause it and free some of its memory.
• Admit fewer requests: lower utilization, wasted bandwidth on weight reads that serve too few sequences, but comfortable memory.

This tension—throughput (total work completed per second) against memory, and utilization against latency (the time one request waits)—is the reason the rest of this book exists. Later chapters ask how many requests may run together, how their growing state should be stored, how work should be divided across devices, and how the service should behave when demand exceeds capacity. All are consequences of one fact: sharing the model makes inference more efficient, while every additional request brings state and delay.

The numbers, collected

For reference, the constants used above and throughout the book’s exercises:

These are planning estimates for a well-tuned system, not promises. Real measurements on real hardware are always the final authority.

Try it yourself

The mechanics described above are not hypothetical. You can observe them on a smaller model with a single GPU.

Start a server

Install vLLM and launch a server with an 8-billion-parameter instruction-tuned model (small enough to fit on one consumer GPU with 24 GB of memory):

pip install vllm
vllm serve meta-llama/Llama-3.1-8B-Instruct --dtype auto

The --dtype auto flag lets vLLM choose a numeric format, such as BF16, that the model and hardware support. The server loads the model weights into GPU memory and begins listening for requests on port 8000.

Send a request

From another terminal, send a prompt and observe the response:

curl -s http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [
      {"role": "user", "content": "Explain what a KV cache is in three sentences."}
    ],
    "max_tokens": 128,
    "stream": true
  }' | head -20

With "stream": true, you will see server-sent events—small messages sent over one long-lived HTTP response—arrive one at a time. Each carries a token or a small group of tokens. The gap before the first event is TTFT: prefill plus any queue wait. The rhythm of subsequent events is the decode cadence.

What to observe

While the request runs, a few measurements connect to this chapter’s content:

1. GPU memory usage. Run nvidia-smi, NVIDIA’s command-line GPU status tool, in a third terminal. Note the memory consumed after model loading (weights plus runtime overhead) and watch whether it changes during generation (KV cache allocation).

2. Time to first token. The delay before the first streamed event includes tokenization, queueing, prefill, and one decode step. Compare a short prompt with a longer one while keeping every other setting fixed.

3. Token generation speed. Count the streamed events per second. Compare this measured rate with the time of one decode step. Do not expect the Atlas estimates to match: this model is smaller and your hardware and software revisions determine the actual result.

4. Concurrent requests. Open several terminals and send requests simultaneously. Watch GPU memory climb (more KV cache) and per-request token rate decline (the same bandwidth now serves more sequences). This is the throughput-memory tension from the scaling section above, made visible.

The 8B model is not Atlas. Its KV cache is smaller, its decode steps are faster, and it fits on one device. But the structure is identical: tokenize, prefill, decode loop, sample, detokenize, stream. Everything observed here scales, with the same tensions, to the 70B model and beyond.

Further reading

You do not need these resources to continue to Chapter 1. Use them when one of the chapter’s new concepts deserves a slower or more visual second explanation.

• Text and tokens: Hugging Face’s tokenizer introduction explains why models consume token IDs and how word and subword tokenization differ.
• Transformers and attention: Google’s illustrated Transformer introduction develops embeddings, attention, and decoder generation visually.
• BF16 and reduced precision: NVIDIA’s TensorRT developer guide describes FP32, FP16, and BF16 and the trade-off between numerical range, precision, memory, and speed.
• Compute-bound versus memory-bound work: NVIDIA’s roofline profiling guide connects arithmetic intensity, memory bandwidth, and peak computation.
• Why KV-cache layout matters: the vLLM PagedAttention paper shows how request state and memory fragmentation limit batching in a production inference engine.

What comes next

This chapter traced one request through one model replica: text to tokens, tokens through a stack of Transformer layers, persistent state created during prefill, a bandwidth-bound decode loop, sampling, and text back out. The path was linear, and the request did not yet compete with other users.

The rest of this book is about what happens when this simple loop must serve thousands of users simultaneously, across many GPUs, under strict latency and quality contracts. Chapter 1 introduces the three planes of decisions — data, control, and management — and the five categories of state that the service must protect. Chapter 2 defines what “fast” and “good enough” mean precisely enough to measure. Chapter 3 returns to the model’s execution in full detail, including mixture-of-experts routing, encoder stages, and diffusion. From there, every chapter zooms into one region of the system that the single-request loop left simple.

The loop does not change. The engineering is in making it work under contention.

1. The Serving System: Decisions, State, and Ownership

Imagine that you send a question to a customer-support assistant. The answer begins streaming half a second later and finishes a few seconds after that. From the outside, one request went in and one answer came out.

Now picture the same assistant on a busy morning. The same question reaches the same model, but the first word appears after four seconds, and the stream stalls twice before finishing. Nothing about the model changed. The weights are identical, the prompt is identical, and the hardware is identical. What changed is the serving system around the model: how much work arrived, how it was placed, and what had to wait.

Inside the service, much more happened than “run the model.” A web server checked the request and applied a chat template. A tokenizer turned text into integers. A router chose a model replica. A scheduler found room beside other requests already in progress. The model read weights and conversation state from GPU memory one step at a time. Another component converted new token IDs back into text and sent them over the network.

Any one of those steps can become the reason the assistant feels slow or fails. Inference engineering is the work of understanding the whole path. This chapter builds the map: the life cycle every request follows, the decisions made at three different speeds, the state that must be protected along the way, and the trap of improving one part while the service gets worse. Every later chapter zooms into one region of this map.

We will build that map from running systems, not from an abstract architecture. The two implementations followed throughout this book are vLLM and SGLang. Their names and process boundaries differ, but both have to turn an incoming request into scheduled GPU work and an ordered stream of output. Start with their code paths; the vocabulary in the rest of the chapter will then name things you have already seen.

Meet the two engines

This edition studies fixed source snapshots so that a link continues to mean the same thing as the prose beside it: vLLM at 5cecfc0 and SGLang at e161bd1. Do not try to memorize either repository. Learn to recognize the same six duties in both: accept a request, prepare it, admit it, schedule it, execute it, and return its output.

A first map of vLLM

The shortest useful route through vLLM begins at its asynchronous engine interface, crosses a process boundary, and ends at the GPU model runner.

A vLLM request crosses three major ownership boundaries.

flowchart LR
    A["OpenAI API and AsyncLLM"] --> B["EngineCore and Scheduler"]
    B --> C["Executor and GPUModelRunner"]
    C --> B
    B --> A

Use this source map on a first reading. The goal is to know where to resume when a trace or metric points at one stage.

Follow one generation request in that order. The API layer calls AsyncLLM.generate. That method’s own documentation describes the handoff: create an output stream, prepare the input, add detokenization state, then submit to an EngineCore running separately. EngineCoreClient carries that message across the process or transport boundary. EngineCore.add_request places the request under scheduler ownership.

The repeated serving loop is visible in EngineCore.step. It asks Scheduler.schedule for the next work, calls the executor’s execute_model, drains cancellation requests, and gives completed model output back to the scheduler. The model runner updates request state and KV block tables before dispatching the model. Results travel in the opposite direction: engine outputs reach AsyncLLM’s background output handler, which feeds the stream belonging to the original caller. A request therefore does not live in one function. It changes owners at explicit seams.

A first map of SGLang

SGLang exposes the same duties through a different split. Its TokenizerManager is a substantial request-side manager, while its Scheduler owns a long-running event loop and communicates with a tensor- parallel model worker.

An SGLang request moves from a frontend manager into a scheduler process.

flowchart LR
    A["HTTP server and TokenizerManager"] --> B["Scheduler and memory pools"]
    B --> C["TpModelWorker and ModelRunner"]
    C --> B
    B --> A

Trace the code from http_server.generate_request. It delegates to TokenizerManager.generate_request, which normalizes the request, creates request state, validates adapter selection, tokenizes input, sends the tokenized object to the scheduler, and awaits responses. In the scheduler, handle_generate_request constructs the internal Req object. The event loop then receives pending work, chooses a batch, calls run_batch, and processes the result.

The model-facing half begins in TpModelWorker.forward_batch_generation. It constructs a ForwardBatch, calls ModelRunner.forward, and samples when the worker owns the final pipeline stage. Back in the scheduler, process_batch_result updates request progress and publishes a load snapshot that a router can consume. Output token IDs pass through DetokenizerManager before the frontend yields text to the HTTP stream. As in vLLM, cancellation, memory release, and output delivery cross several owners; closing the network connection cannot safely erase them all at once.

The same duties, different boundaries

The comparison is more useful than either directory tree alone:

Neither arrangement is the universal architecture. The important fact is that both must assign the same duties and preserve state while ownership changes. The rest of this chapter gives those duties portable names. Later chapters return to these exact files and descend one level at a time.

From model call to serving system

It is useful to begin with a simple request life cycle:

One request crosses several queues and state owners.

flowchart LR
    A["Client"] --> B["API and validation"]
    B --> C["Router"]
    C --> D["Engine queue"]
    D --> E["Scheduler"]
    E --> F["Model runner"]
    F --> G["Output stream"]
    G --> A
    H["KV and session state"] <--> E

receive -> validate -> prepare -> wait -> execute -> stream -> finish

Each stage has its own concern, and each can fail independently of the model.

Receiving and validating establishes what the request even is: which model, which limits apply, whether attachments parse. Doing this before any expensive work means a malformed request costs microseconds, not GPU time. Preparation turns the caller’s text into the model’s input: a chat template inserts role markers, and a tokenizer produces integer IDs. Given fixed tokenizer and template revisions, this stage is deterministic, which is why later chapters treat those revisions as part of the served contract.

The waiting stage matters because production requests rarely get a GPU to themselves. A request that would execute in 200 ms may spend a second in a queue ahead of it. It helps to notice that there is rarely one queue. An edge queue forms in front of validation, an engine queue forms in front of admission, and a distributed deployment adds queues between stages — a completed prefill may wait for decode capacity on another worker. Queue position is decided by admission and scheduling policy, which is why Chapters 6 and 16 own most of the latency story, and why a trace that records only one queue length explains only part of every wait.

Execution may happen over hundreds of short model steps rather than one call. The shape of the work also changes along the way. For a language model, the initial prompt is processed as a group during prefill: a 1,000-token prompt means roughly 1,000 positions computed together. New output tokens are then produced one at a time during decode: each step adds a single position per active conversation. Prefill and decode use the same weights but stress the hardware differently, and Chapter 3 builds the machinery for saying precisely how.

Streaming makes partial progress visible while decode continues, and finishing releases request state or retains some of it for future reuse. Retention is a decision, not an accident: kept state can serve the next turn or the next user, and Chapter 7 formalizes when reuse is safe.

A vision-language request adds image decoding and a vision encoder before the language model. An image-generation request runs a denoising model many times instead of producing tokens at all. The stages survive, but their costs move. For this reason, a request is better understood as a small workflow than as one model invocation.

The precise single-request timeline belongs to Chapter 0. At this level, retain only the ownership map: the API owns validation, the router owns placement, the engine owns admitted work, and the output path owns ordered delivery. Load changes the time spent at those boundaries without changing the model itself. Chapter 2 names the resulting latency populations; Chapters 5 and 6 implement the engine boundaries and scheduling decisions.

Three kinds of decisions

As the workflow runs, the service makes decisions at different speeds.

The three planes operate at different time scales but share evidence.

flowchart TB
    M["Management plane: deploy and configure"] --> C["Control plane: place and recover"]
    C --> D["Data plane: schedule and execute"]
    D --> T["Metrics, logs, and traces"]
    T --> C
    T --> M

The first diagram follows one request left to right. Each arrow crosses an ownership boundary: the client owns nothing after send, the API owns the validated request, the router owns the placement decision, and the engine owns everything from admission onward. The state box hangs off the scheduler rather than the model runner because conversation state outlives any single step; the runner borrows it for the duration of one batch.

The second diagram lifts the view. Requests flow through the data plane, while measurements flow back out of it. When something breaks, two questions come before any fix: which plane owns the failing decision, and which kind of state was being read or written?

The data plane makes immediate decisions about current requests. It chooses the next batch, allocates memory, launches model work, samples outputs, and streams events. These decisions happen many times per second, and their inputs must already be in memory: a data-plane decision that waits on a network call stalls every conversation sharing the step.

The control plane decides where work should go. It routes requests between replicas, tracks which prefixes are cached, changes membership when a worker fails, and tells overloaded services to stop accepting more traffic. Its view is broader than one model step, but it still reacts to live conditions, usually within milliseconds to seconds.

The management plane changes the service itself. Deploying a new model, rotating credentials, changing capacity, and rolling out a new engine version belong here. These events are rare relative to request traffic, but they redefine what correctness means for everything running underneath.

Separating these decisions prevents a common design mistake: making a decision in the wrong place. A decode step that consults a global database has moved a control-plane question into the data plane, and every user feels the stall. Conversely, a deployment that ignores resident caches has moved a correctness question into the management plane, where it is easiest to miss. State created by old weights must not be reused with new weights, and only coordinated rollout can guarantee that. The time scales differ, but correctness connects the planes.

Most operational incidents are legible through this lens. A router that keeps preferring a warm replica is a control-plane decision missing data-plane evidence. A fleet that slowly drifts across versions is a management-plane process missing enforcement. Naming the plane is often the first step toward the fix.

What connects the planes is evidence. Metrics, logs, and traces are produced by the data plane, aggregated for the control plane, and summarized for the management plane; each consumer needs a different resolution of the same events. This is why observability, Chapter 24’s subject, is not a feature added after the fact but the shared currency that lets three decision speeds coordinate without sharing a fate.

Follow the state

Component diagrams show where code runs. State tells you what the system must protect.

Return to the support assistant. The model weights are long-lived and mostly immutable. The request text, deadline, and generated tokens belong to one request. Intermediate activations exist for only part of a model step. The attention keys and values created from the conversation may live for the whole request and remain useful for the next turn. Queues, worker health, and cache locations describe the service as a whole.

These lifetimes suggest five broad categories:

For any important state object, ask who creates it, who may change it, how a consumer recognizes the correct version, and what happens when the owner fails. Those four questions uncover many bugs before code does.

The four questions applied to one object per category

The questions earn their keep on concrete objects. Take one representative from each category.

The weight files (model state) are created by the deployment pipeline and changed only by a management-plane rollout. Consumers recognize the correct version because the version travels with the deployment, not because a worker happens to hold recent bytes. If the owning worker fails, recovery must reload the same version; a replacement worker that fetches “current” weights after a mid-rollout crash can join the fleet holding different weights than its peers.

The request record (request state) is created at admission and mutated by the data plane as steps complete. Its identity is the request ID, and every downstream event must carry that ID or it cannot be attributed. If the engine crashes, the record dies with it: the client experiences a disconnect, and whether a retry is safe becomes an API contract question, which Chapter 22 develops.

An activation workspace (execution state) exists for part of one step and is owned by the model runner. Nobody else may read or write it, and it is freed when the step ends, success or not. Memory leaks in serving systems usually mean someone broke this lifetime: a buffer allocated per step but released conditionally.

A KV block (reusable state) is created during prefill and may outlive its request. Identity is subtle: a block can be shared by branches of one conversation, and whether it can be shared across requests depends on content and context, which Chapter 7 owns. Physical memory is freed only when the last reference disappears. The classic failure is freeing a block while a GPU step still reads it, which corrupts an unrelated request’s output.

A queue entry (service state) is owned by the control plane and changes continuously. Its characteristic failure is staleness: a dead worker remains listed as healthy until a check fires, and the router keeps sending traffic to a destination that cannot answer. Freshness requirements, not importance, distinguish service state from model state.

Cancellation shows how the categories interact under pressure. Closing the network stream does not erase work already scheduled on a GPU. The service must stop scheduling future steps, decide what to do with work already in flight, release memory exactly once, and send a final protocol event if the connection still exists. Each obligation lands on a different category: future steps touch request state, in-flight work touches execution state, memory release touches reusable state, and the final event touches the request record again. Cancellation is therefore a state transition, not merely an HTTP feature.

The execution plan

An inference service needs a plan for placing and advancing work. The plan answers questions such as these:

• How many requests can share a step? This sets throughput against per-request latency and is answered by the scheduler every step.
• Which devices hold each part of the model? A capacity and topology question, fixed at deployment and revisited in Chapters 4 and 12.
• How much memory is reserved for request state? Reservation protects interactive traffic from bulk work, and getting it wrong shows up as preemption storms.
• Which input shapes use compiled graphs? Capture trades flexibility for launch cost, and the shapes that actually arrive decide whether the trade pays.
• When should state be reused, moved, or discarded? Reuse saves computation but creates coupling between requests.
• Which traffic receives priority during overload? Overload is normal, not exceptional, and the priority policy is a product decision.

The best answers depend on the workload. A chat service with a long shared system prompt benefits from prefix reuse. A batch summarization job may care more about total completion time than first-token latency. A real-time voice assistant values steady output and fast cancellation. There is no useful configuration without a workload and a service objective.

The same plan questions, asked of three different services, produce different plans — which is the strongest argument against copying configurations:

Reading the table by column shows each service committing to a coherent posture; reading it by row shows that no single answer wins. A configuration is only defensible next to the workload it serves.

Measurements complete the process. The service observes queues, latency, throughput, cache use, failures, quality, and cost. Those observations show whether the plan should change.

workload and goals
       |
       v
model + hardware -> execution plan -> measured service behavior
                          ^                    |
                          +---- revise --------+

This feedback loop is the organizing idea for the rest of the book. Each part narrows it: single-engine mechanics first, then distribution, then new modalities, then the discipline of operating the loop in production.

Why faster parts can create a slower service

Suppose a team captures the model in a GPU execution graph and saves a small amount of launch overhead on every decode step. The captured graph expects a fixed batch shape, so the engine pads small batches to a much larger size. At low traffic, the GPU now performs enough extra work that users wait longer.

Both statements are true: graph replay made each prepared operation cheaper, and the service became slower.

Counting a padded replay

The paradox becomes clear with small numbers. Assume a measured step time of 5.2 ms of GPU work for a batch of three decoders running eagerly, plus 0.9 ms of CPU launch gaps between kernels, for 6.1 ms wall time per step. Assume also that the engine has captured graphs at batch sizes 1, 2, 4, and 8, and that a captured step pads its tensors to the bucket size.

With three active requests, the engine selects the size-8 bucket. Padding adds five idle rows to every batched operation. Suppose the padded step performs 6.4 ms of GPU work, of which 1.2 ms belongs to the padding, and the captured graph collapses launch gaps to 0.2 ms, for 6.6 ms wall time.

The local optimization worked exactly as advertised: launch overhead fell from 0.9 ms to 0.2 ms. The service still regressed, because the step got slower overall: 6.6 ms instead of 6.1 ms. Across a 200-token response, that is roughly 100 ms of added time to first finish, paid by every user during every quiet period, in exchange for launch savings that mattered only when the GPU was already saturated. Whether the trade is right depends on the traffic distribution, which is why Chapter 9 treats bucket selection as a workload question and Chapter 23 insists on measuring it end to end.

The same tension appears in many forms. Cache-aware routing can overload the replica with the best prefix. Large prefill chunks can improve GPU efficiency while interrupting active decoders. Tensor parallelism can reduce arithmetic per device while adding a network synchronization to every layer. A more compressed weight format can save memory but use a slower kernel for the shapes that actually arrive. Each of these reappears, with its own arithmetic, in Chapters 16, 6, 12, and 10.

The lesson is not that local optimization is bad. It is that the unit of success is the service objective under a realistic workload.

Worked example: the cache hit that loses

Suppose a 6,000-token document question reaches a router. Replica A already has 4,000 tokens of the document cached but has 450 ms of queued prefill work. Replica B is idle and can recompute those 4,000 tokens in 240 ms. A router that sees only cache locality sends the request to A and adds at least 210 ms to the user’s wait.

Walk the comparison. On replica B, the user waits the full recomputation: 240 ms of prefill before their own tokens begin producing output. On replica A, the user first waits behind 450 ms of work that arrived earlier. Even if the cached prefix reduced replica A’s remaining work to nearly nothing, the queue alone exceeds replica B’s entire prefill by 210 ms. The cache saved computation that was not the bottleneck; the queue was.

Cache value is therefore conditional on queue state, and a locality score that ignores queues is not a conservative approximation — it is a different decision. Chapter 17 builds routers that weigh both.

The useful trace is not merely router -> worker. It records the router’s queue estimate, matched-token estimate, decision time, and the worker that became authoritative. At the worker it separates admission wait, allocation, prefill, decode, and output buffering. With that trace, the wrong choice is visible as data: a large matched-prefix count paired with a larger admission wait than an idle alternative.

Ownership explains why the trace needs both ends. The request record lives with the router and then the engine, while the KV blocks live on whichever replica becomes authoritative. When the request is cancelled mid-flight, cancellation must reach both owners without freeing blocks that a GPU step still reads — the exact discipline the state categories imposed earlier.

This example gives the trace a purpose: explain why the apparently valuable cache hit made the service slower. Chapter 2 will turn that observation into a goodput and latency comparison.

Practice: produce an ownership trace

Trace a 6,000-token document request with a 300-token output limit through edge queue, validation, tokenization, routing, engine admission, prefix lookup, prefill, decode, detokenization, and streaming. For every boundary, record the queue, state owner, cancellation behavior, and one timestamp.

A useful artifact is a table with one row per boundary and one column per recorded property, so that any row with an empty owner cell marks a state object nobody is responsible for releasing. Then compare a replica with a 4,000-token match and 450 ms queue against an idle replica that recomputes the prefix in 240 ms. State which replica you choose and which two metrics would reveal a wrong choice in production. A worked answer is in Appendix G.

2. Workloads, SLOs, and Goodput

Two teams benchmark the same model on the same GPU. One reports 20,000 output tokens per second. The other reports that 95 percent of users see a first token within 400 milliseconds. Which system is faster?

The numbers answer different questions. The first describes how much work the server completed. The second describes how the service felt to most users. Neither is sufficient on its own, and each can be made to flatter a failing system: token counts rise when batches grow large enough to hurt latency, and a latency percentile improves when slow requests are quietly excluded.

Before tuning an inference system, you need a precise description of the work that arrives and the promises the service must keep. Otherwise, a benchmark can improve while the product gets worse. This chapter builds the vocabulary that makes such comparisons impossible to fake: units of work, the several latencies hiding inside “how long did it take,” percentiles and how to combine them, capacity as distinct from throughput, goodput as the SLO-qualified rate, and the workload description that all of it depends on.

Decide what counts as work

A text-generation service handles several nested units. A session contains turns. A turn may produce one request. A request can create several candidate sequences, and every sequence contains input and output tokens.

Media systems use different units: images, frames, audio chunks, latent patches, or generated samples. Reinforcement-learning systems also group generations by prompt and policy version.

This is why “requests per second” and “tokens per second” need qualifiers. A request containing 50 input tokens is not the same job as one containing 50,000. Token throughput may count input tokens, output tokens, or both. It may even include padded positions or speculative tokens that were later rejected — work the hardware performed that no user received.

Retries create the same ambiguity at the request level. When a client times out and sends the request again, the second attempt is new work to the server but not to the user; a service that counts attempts reports higher volume precisely when it is failing more often. The same accounting question survives into goodput: if both attempts complete but only within-SLO attempts qualify, the definition must say whether the retry’s cost lands in the denominator. Chapter 22 treats retry identity as an API contract for exactly this reason — the metric story and the correctness story are the same story.

A useful metric always names its unit. For example:

The service completed 320 successful requests per second, where requests had the production input and output length distribution.

That sentence is less impressive than a large unqualified number, but far more useful. It is also worth treating metric definitions as part of the service’s contract, with the same discipline as any interface: written down, versioned, and changed deliberately. When a dashboard redefines “latency” or a release switches which tokens enter the numerator, every trend line built on the old definition breaks without anything in the system failing. Chapters 21 and 22 return to this bookkeeping as an engineering obligation, not a reporting courtesy.

Unit choice does not merely add precision — it can reverse a ranking. Assume service A completes 100 requests per second at 500 output tokens each, and service B completes 200 requests per second at 100 output tokens each. On requests per second, B wins two to one. On output tokens per second, A wins 50,000 to 20,000. Both numbers are correct; they describe different products. A caller chaining generations downstream cares about A’s token rate, while a caller issuing short classifications cares about B’s request rate. Declaring the unit is therefore part of declaring the audience.

Latency is not one number

Consider a response that streams ten tokens. The first token appears after 600 milliseconds. Most later tokens arrive 40 milliseconds apart, but one gap lasts half a second.

End-to-end latency tells you when the response finished. Time to first token (TTFT) captures the initial wait. Inter-token latency (ITL) captures each gap in the stream. Time per output token (TPOT) averages the time after the first token across the remaining tokens:

TPOT = (end-to-end latency - TTFT) / (output tokens - 1)

TPOT is compact, but it can hide the half-second pause. When output cadence matters—as it does for chat, code completion, or speech—report ITL percentiles or count stalls above a product threshold.

It also helps to break total latency into the stages a team can act on:

network ingress
  + queueing
  + preprocessing
  + model execution and intermediate transfers
  + postprocessing
  + network egress

If a benchmark starts its timer after queueing and stops before streaming, it does not measure the user’s latency.

Attributing a slow first token

The stage stack earns its place when it turns a complaint into an action. Suppose users report that first tokens sometimes take over 600 ms, and traces from one such request decompose the wait as follows: network ingress 10 ms, edge queue 40 ms, engine queue 310 ms, preprocessing 15 ms, prefill 205 ms, and egress 20 ms — about 600 ms in total.

Each stage names a different owner and a different remedy. The two queue terms dominate at 350 ms combined, so no model or kernel work will fix this request; the levers are admission policy, chunk sizing, and routing, which are Chapters 6 and 16 subjects. Prefill’s 205 ms is real computation, and shortening it means prefix reuse or hardware, not scheduling. The remaining 45 ms of ingress, preprocessing, and egress is already near irreducible floor. Attribution prevents the classic misresponse: tuning attention kernels for a week because “the model felt slow,” while the request spent half its life waiting in a queue no dashboard displayed.

The decomposition also defines what a trace must record to be useful later. A timestamp at each stage boundary costs microseconds at admission time and is impossible to reconstruct afterward from end-to-end totals alone.

Why a per-token p99 is not a per-request p99

Percentile claims inherit the population they are computed over, and token gaps and requests are different populations. A small example makes the gap impossible to ignore.

Suppose two requests are observed. Request A streams eleven tokens with gaps of 40 ms each. Request B streams two tokens, and its single inter-token gap lasts 500 ms. Pool all eleven gaps together and exactly one of them exceeds 400 ms: the token-gap p99 sits near 500 ms, but the token-gap p90 is a comfortable 40 ms. Now count by request instead: one of the two requests contained a half-second stall, so half of the users experienced it. No per-token percentile below the extreme tail can express that.

TPOT averages the same evidence differently still. Request A contributes ten 40 ms gaps; request B contributes one 500 ms gap. The token-weighted average is about 80 ms, because A’s many well-behaved tokens outnumber B’s single bad one. The user-weighted story is that every second response stalled. None of these numbers is wrong; each answers a different question. The failure mode to avoid is quoting whichever one flatters the system, which is why the reporting rules in the next section demand that the population be stated every time.

Why percentiles matter

An average combines ordinary requests with rare slow ones. In production, those slow requests may be the exact cases a customer remembers.

The 99th percentile, or p99, is the value below which 99 percent of observations fall in a stated population and time window. The population matters. A global p99 can hide a small tenant that is consistently slow. A per-token p99 is not a per-request p99. A number calculated from successful requests says nothing about timeouts that were removed from the sample.

When reporting a percentile, state:

• what was observed: request, token gap, or session;
• which traffic was included;
• the test or production window; and
• how errors and cancellations were handled.

A well-formed claim reads like: “p99 of per-request TTFT, all tenants including retries, measured at the edge over 09:00–10:00, cancellations counted as failures.” Every clause removes one way to misread the number, and omitting any clause leaves the reader to guess — usually generously.

Do not average percentiles produced by separate hosts. Combine the underlying samples or merge compatible histograms, then calculate the percentile.

Merging histograms without lying

Production services report latency histograms per host, so combining them is an everyday operation with two honest requirements: bucket boundaries must match, and resolution must be reported honestly.

Assume two hosts expose request-latency histograms with boundaries at 100, 200, 400, and 800 ms. Host A served 100 requests with counts [90, 8, 2, 0] per bucket; host B served 100 with counts [95, 3, 1, 1]. Merging is simple addition per bucket: [185, 11, 3, 1] over 200 requests. The merged p50 falls in the 100–200 ms bucket, since the 100th ordered observation lands there. For the merged p99, the 198th of 200 observations falls in the 400–800 ms bucket, so the honest statement is that p99 lies between 400 and 800 ms — not a point value. If each host had exported finer buckets, the merged estimate would tighten accordingly.

Both failure modes are now visible. Averaging the hosts’ individual p99 values instead of merging would produce a number no user experienced. And merging histograms with different boundaries silently fabricates precision: the counts cannot be added because they describe different intervals. When boundaries disagree, the only correct path is back to raw samples. Bucket width bounds percentile resolution forever, which is a reason to choose histogram layouts deliberately rather than accept a default.

The window deserves equal care. A p99 over a rolling five-minute window and a p99 over the full day describe different services: the daily figure blends the quiet night in and can hide an hour of morning degradation entirely, while the short window surfaces it but also flatters any moment that happens to follow a quiet stretch. Deployments add their own trap — a host that joined mid-window contributes partial data unless its coverage is recorded. State the window with the percentile, and when two windows disagree, treat the shorter one as the more urgent message rather than the noisier one.

Throughput is not capacity

Throughput measures completed work per unit time. Capacity is the arrival rate the service can sustain while meeting its contract. The two diverge near overload.

Goodput filters completed work through the product contract.

flowchart LR
    A["Arrivals"] --> B["Queue"]
    B --> C["Inference service"]
    C --> D["Completed requests"]
    D --> E{"Meets latency, quality, and correctness SLO?"}
    E -->|Yes| F["Goodput"]
    E -->|No| G["Completed but non-qualifying work"]

Imagine a server completing 100 requests per second while 120 arrive. Its throughput looks stable, but the queue grows by 20 requests every second. Latency will continue rising until callers time out or the system fails.

Appendix A collects a useful sanity check here. With λ the arrival rate, W the average time in the system, and Q the average number of requests present, Little’s Law says Q = λ · W. In the overloaded service, completion lags arrival by 20 requests per second, so Q climbs without bound: after thirty seconds, roughly 600 requests are waiting whose owners have not yet noticed. Running the law in the diagnostic direction is just as valuable — measure any two of the three quantities and the third is determined, so a dashboard showing stable W while λ rises must also show Q rising somewhere, and if it does not, one of the measurements is lying about its population. The law speaks in averages and assumes a stable system; it predicts nothing about tails, which is precisely why percentiles exist alongside it.

This leads to goodput: the rate of work that satisfies the service-level objective, or SLO.

request goodput = qualifying completed requests / test duration

A request might qualify only if it returns without error, begins within 500 milliseconds, maintains acceptable output cadence, and produces a valid result. For a structured-output endpoint, malformed JSON does not count as goodput even if it arrived quickly.

Finding capacity by search, not assertion

Capacity is a property of the workload plus the contract, so it is found empirically: offer increasing load and watch where qualifying work stops keeping up. Assume an open-loop generator offers 60, 80, 100, and 120 requests per second against the same service, and the SLO-qualified completion rates come back at 60, 79, 94, and 72.

The service’s capacity for this workload is roughly the offered rate where goodput peaks — here near 100, where qualifying work is still rising but the margin has vanished. Beyond the knee, extra offered load does not add completed work; it displaces it, because arrivals spend longer in queues and miss latency conditions they would have met at lower load. Note what the experiment does not claim: the knee moves with the workload mix, the prefix correlation, and the SLO clauses, which is exactly why “the service does 100 requests per second” is incomplete until its workload and contract travel with it.

Why the knee sits below 100 percent

The knee’s position is not a policy choice; it falls out of utilization arithmetic. Let ρ (Appendix A) be offered work divided by service capacity. For random arrivals, the simplest queueing model — one shared queue, exponentially spaced arrivals, exponential service — puts average waiting at roughly ρ / (1 − ρ) service periods. The assumptions matter (real serving is batched, correlated, and bimodal between prefill and decode), but the shape of the curve survives every correction: waiting is proportional to ρ near zero and diverges as ρ → 1.

Walk it in Atlas units. Suppose the knee experiment above found capacity near 100 requests per second, so one request occupies the system about 10 ms of exclusive service time on average. At ρ = 0.5 the model predicts about 0.5 / 0.5 = 1 period of waiting; at ρ = 0.8 about four periods; at ρ = 0.9 about nine. Going from half-used to 90-percent-used multiplies queue delay roughly ninefold while raising throughput only 80 percent — and since TTFT includes that delay, goodput collapses long before throughput does, which is exactly what the 120-row in the table showed. This is why operating targets sit at modest utilization: the last 20 percent of capacity costs more latency than it returns work, and headroom is what absorbs arrival bursts. Any interviewer asking “why not run hotter?” is really asking whether you can derive this curve and name where its simplifications break.

Goodput often changes the winner in an architecture comparison. A large batch

The qualification clause is doing real work, so it deserves the same care as the rate itself. Latency-qualified goodput, as above, is only one variant: a structured-output endpoint qualifies on schema validity, a retrieval endpoint on relevance thresholds, a media service on frame deadlines. Token-level goodput — output tokens from qualifying requests — matters when callers compose your service into longer pipelines, because downstream work scales with tokens received, not requests observed. Whatever the clauses, they must be written next to the number; “goodput” without its qualification is just throughput wearing a better name.

Closed-loop load is a thermostat

The closed-loop generator deserves arithmetic, because its self-throttling behavior hides overload from unwary benchmarks. Consider a fixed population of 40 browser clients, each sending one request at a time. By Little’s Law rearranged, the offered rate is Q / W with Q fixed at 40: if a round trip averages W = 0.5 seconds, clients collectively offer 80 requests per second.

The load generator changes what overload looks like.

flowchart TB
    O["Open-loop source"] -->|independent arrivals| S1["Server"]
    S1 --> Q["Queue can grow"]
    C["Closed-loop clients"] --> S2["Server"]
    S2 --> R["Responses"]
    R -->|permit next request| C

The first diagram is a filter, not a pipeline: completed work leaves the service either way, and only the contract decides what counted. The second diagram shows why the same server can produce two different overload stories. An open-loop source keeps sending at its own rhythm while queues grow; a closed-loop client cannot send its next request until the last one returns, so rising latency silently throttles the load. Every measurement later in this chapter is shaped by which of those two worlds produced it.

Now suppose the server degrades until W rises to 2 seconds. The client population has not changed, yet offered load falls to 20 requests per second. Queues drain, the server stabilizes, and measured latency settles at a value that looks acceptable. The benchmark concludes the service survived; in reality it collapsed to a quarter of its usefulness and the missing 60 requests per second are simply clients waiting for permission to speak. An open-loop source offering 80 requests per second regardless of responses would have exposed the same failure as unbounded queue growth within seconds.

Neither behavior is wrong — real browsers are closed-loop, so the effect is physically real. The error is reading a closed-loop result as if it described a capacity. State which loop produced a number, and treat closed-loop “stability under overload” as the thermostat working, not the server coping.

Describe the workload as a distribution

Suppose the support assistant receives mostly short questions. Ten percent of users attach long documents, and half of all requests share one of a few system prompts. Traffic is quiet overnight and arrives in bursts at the start of the workday.

An average prompt length loses most of that information. A useful workload record keeps the distributions of arrival time, input length, output length, modality, media size, priority, tenant, and reusable prefix. It also preserves correlations. Long documents may lead to long answers. Requests with the same prefix may arrive together. Sampling each column independently creates a trace that never existed — a synthetic morning where every long document gets a short answer and prefix-sharing groups dissolve into unrelated traffic, so the cache behaves in ways production never will.

Burstiness belongs in the record for the same reason. Total daily volume can be identical between a flat day and a bursty one while peak queue depth differs by an order of magnitude, and capacity decisions made against the average will fail on the peak.

A workload record should also be replayable: timestamps, lengths, modality, tenant, and prefix identifiers in a form a load generator can consume directly. Replaying the same record against two engine revisions turns “the new version feels faster” into a controlled comparison, and replaying a recorded incident reproduces the arrival pattern that caused it. Chapter 23 builds its benchmarking discipline on exactly this foundation — without a replayable description of the work, every performance claim is an anecdote.

Correlation is easiest to see in prefix sharing. Picture ten conversation groups, each anchored by an 8,000-token system prompt. If requests from a group arrive while its prefix is still resident, one prefill serves every request that follows; if the same requests arrive scattered through the day, the cache evicts between visits and each group’s prefix is recomputed on arrival. The input-token totals are identical — the correlated trace may prefill 8,000 tokens where the dissolved one prefills hundreds of thousands. Any capacity conclusion drawn from token totals alone is wrong in both directions unless the arrival correlation traveled with the data.

Two load-generator styles answer different questions. An open-loop generator sends work according to an external arrival process even when the server slows. It exposes queue growth and overload. A closed-loop generator waits for a response before sending the next request from a client. It models a fixed client population, but as the thermostat example showed, it also reduces offered load automatically when latency rises.

Neither style is universally correct. The mistake is failing to say which one produced the result.

Online, offline, and pipeline-driven work

Arrival semantics change the optimization target even when the model and token counts are identical.

flowchart LR
    O["Online callers"] -->|"independent deadlines"| S["Shared inference capacity"]
    B["Offline dataset"] -->|"completion deadline"| S
    P["Pipeline stage"] -->|"backpressure from consumer"| S
    S --> I["Interactive goodput"]
    S --> D["Dataset completion time"]
    S --> F["Pipeline freshness"]

An online service receives independently timed requests and is judged by per-request latency and availability. An offline job owns a finite dataset and usually trades individual latency for total completion time, accelerator occupancy, and restartability. A pipeline-driven service—an embedding stage feeding retrieval, or a rollout fleet feeding a trainer—receives work at a rate coupled to the consumer. Backpressure and freshness are part of its contract.

Batch inference is not simply online inference with a large concurrency value. It can reorder examples, checkpoint progress, group shapes aggressively, and retry failed shards without preserving a user-visible stream. Conversely, a pipeline cannot maximize batch size blindly when doing so starves its consumer. Record workload mode alongside lengths and arrival distributions so later chapters optimize the correct clock.

Start from the product

Different products need different contracts.

An interactive assistant cares about TTFT, output cadence, cancellation, and tail latency. An offline summarization job may accept high per-request latency if a dataset finishes before a deadline. An embedding endpoint cares about batch throughput and bounded completion time. A real-time media service has frame or audio-chunk deadlines. A rollout service is coupled to a trainer and may value policy freshness alongside generation speed.

Quality and correctness belong in every case. Quantization that meets the latency target but damages an important task is not a successful optimization. A tool call with the wrong schema is not useful output. A better service balances several objectives rather than maximizing one:

quality, correctness, availability, latency, goodput, cost, and energy

Select the model with the system in view

Model selection is often presented as a leaderboard lookup. Inference engineering turns it into a constrained product experiment.

Begin with an evaluation set drawn from actual product work: ordinary cases, high-value cases, adversarial inputs, long contexts, required languages, tool calls, and refusal behavior. Decide which failures are disqualifying before comparing models — deciding afterward, once a favorite has emerged, converts the evaluation into advocacy. The disqualifying set is product-specific: for a coding agent, a tool call that writes to the wrong file is disqualifying no matter how fluent the prose around it; for a brainstorming assistant, the same mistake might be an ordinary quality ding. Then measure every viable candidate behind the serving stack you could realistically operate.

A model with a higher offline quality score may be a worse product choice if it misses the interaction deadline or requires a topology the team cannot keep reliable. A smaller model may be preferable if retrieval supplies the missing knowledge. Fine-tuning can change behavior without solving serving cost; distillation can change both behavior and the execution envelope. Quantization may let a candidate fit on fewer devices, but only an application evaluation can determine whether its numerical changes are acceptable.

Write a one-page selection record for each serious candidate:

• the exact model, tokenizer, precision, context limit, and license;
• quality results on the product evaluation set;
• memory fit and required accelerator topology;
• latency and goodput under the target workload;
• operational dependencies and fallback behavior; and
• expected cost at ordinary and peak traffic.

The record prevents a common reversal: choosing a model in isolation and later discovering that the service contract cannot afford it. Model and system selection are one decision viewed at two levels.

Worked example: throughput without goodput

One hundred requests arrive over 12.5 seconds. The server completes 96, so its request throughput is 7.68 requests per second. Only 81 begin within 600 ms, avoid token gaps above 150 ms, finish successfully, and return valid output. Goodput is therefore 6.48 requests per second.

Walk the accounting. Completed work divides into three groups. Eighty-one requests satisfy every clause of the SLO and count toward goodput. Fifteen completed successfully but missed a latency condition — perhaps their TTFT landed at 700 ms behind a burst of prefills. Four errored or timed out before completing. Throughput counts the first two groups: 96 divided by 12.5 gives 7.68. Goodput counts only the first: 81 divided by 12.5 gives 6.48. The fifteen-point spread between those rates is the entire content of this chapter expressed as a number.

Removing the fifteen slow-but-completed requests from the latency sample would make the percentile look better and destroy the meaning of the SLO. They must remain completed work that failed to qualify. The four requests that errored or timed out also remain in the workload accounting.

Now keep total tokens fixed but move arrivals into five bursts. Arithmetic work is unchanged, yet queues form ahead of each burst and long prefills collide with active decodes, so more requests miss the TTFT and ITL conditions and goodput falls even though the server performs the same number of token operations. The per-request traces make the mechanism visible: qualifying failures cluster in the seconds after each burst begins, when queue depth peaks. This is why the arrival process and length correlations belong in the workload definition.

Practice: construct comparable traces

Create three 100-request traces, each containing 100,000 input and 20,000 output tokens: evenly spaced uniform requests, five bursts with mixed lengths, and ten conversation groups sharing 8,000-token prefixes. Use an open-loop rate of 8 requests/s, then repeat closed-loop.

Report queue time, TTFT, worst per-request ITL, end-to-end latency, prefix matches, errors, throughput, and the SLO-qualified goodput defined above. Explain why equal token totals do not imply equal capacity. See the worked construction in Appendix G.

3. Model Topologies as Execution Graphs

When a language model writes a sentence, it does not plan the whole sentence and reveal it one word at a time. It repeatedly predicts what should come next. That simple loop shapes almost every part of an LLM server.

An engine does not schedule an abstract “model.” It schedules operations with particular tensor shapes, dependencies, and state. A scheduler that treats every request as identical work will pack batches badly, reserve the wrong memory, and pick graph shapes that never occur. To understand the engine, we first need to understand the work the model creates.

This chapter inventories that work: the autoregressive loop and its serial dependency, the two kinds of passes a decoder performs, the persistent state attention requires, the irregularities that mixture-of-experts routing, encoders, and diffusion introduce, and the way all of it becomes a serving topology. The emphasis falls on execution properties an engine must discover, rather than one fixed block diagram, because serving-oriented architectures have changed many details since the original Transformer described in Attention Is All You Need.

From a token loop to a work graph

Chapter 0 followed the token loop end to end. Here the loop is useful only as a dependency graph: each decode position reads weights and retained model state, produces logits, commits one sampled token, and makes the next position eligible. A sequence is serial across positions even when thousands of sequences execute together.

That graph exposes the quantities a server must schedule: tensor shapes, persistent bytes, conditional branches, collective communication, and independently placeable stages. The rest of this chapter compares model families through those quantities rather than repeating tokenization, sampling, and detokenization.

One model, two kinds of work

The first pass over the prompt is called prefill. The model processes many input positions at once and creates the attention state needed later. Large matrix operations during prefill tend to use the accelerator’s compute units well: a 1,000-token prompt performs a thousand positions’ arithmetic while reading the weights once, so its intensity resembles ordinary training-style compute.

Prefill creates persistent state; decode consumes and extends it.

flowchart LR
    P["Prompt tokens"] --> F["Prefill"]
    F --> K["KV state"]
    K --> D["Decode step"]
    D --> L["Logits and sampling"]
    L --> N["Next token"]
    N --> K
    N --> D

After prefill, the model enters decode. Each active sequence usually adds one position per step. At a small batch size, the GPU repeatedly reads a large set of weights to do relatively little arithmetic. Decode is therefore often limited by memory traffic or launch overhead.

Batching more sequences lets the same weight read serve more work. That raises throughput, but a request may wait longer for its place in the batch. The scheduler spends much of its life balancing this exchange between hardware efficiency and user latency.

Prefill and decode use the same weights, yet behave like different workloads: compute-bound versus bandwidth-bound, large regular shapes versus thin ones, one burst versus a long cadence. Later chapters will use that fact repeatedly to motivate chunked prefill, separate graph shapes, phase-specific parallelism, and disaggregated serving. None of those mechanisms would exist if the two passes had the same execution profile.

The last row is the scheduler’s dilemma in miniature: prefill work can be sliced into chunks and interleaved, but a decode step is atomic — every resident sequence advances or none does. That asymmetry, not hardware preference, is why phase-mixing decisions dominate Chapter 6’s design space.

Attention remembers the past

Inside a transformer block, attention lets each position combine information from earlier positions. Recomputing the entire prompt for every new token would be wasteful — output token ten thousand would re-derive ten thousand positions’ intermediate results. Instead, the model stores the keys and values created for previous positions. This persistent state is the KV cache.

For a conventional attention layout, a rough size estimate for one sequence is:

KV bytes = 2 * layers * tokens * KV heads * head dimension * bytes per value

The factor of two accounts for keys and values. The total grows with sequence length and can become much larger than the temporary activation memory of one decode step.

The cache formula, applied

Numbers make the formula’s consequences concrete. Take the dense decoder this chapter inventories: 80 layers, 8 KV heads, head dimension 128, BF16 values of 2 bytes each. One token, one layer:

2 * 1 * 8 * 128 * 2 = 4,096 bytes per layer per token

Across 80 layers, each token accumulates 320 KiB of state. An 8,000-token conversation therefore holds about 2.44 GiB — larger than many models’ entire weight footprint was, not long ago. Two scheduling consequences follow directly. First, admission decisions are memory decisions: accepting one more long-context conversation commits gigabytes, not megabytes. Second, the state’s growth is linear in context, so a service whose users drift toward longer conversations watches its effective capacity shrink even though nothing changed.

Architecture choices move the constant by large factors. Grouped-query and multi-query attention use fewer KV heads — eight instead of thirty-two shrinks the cache fourfold. Multi-head latent attention stores compressed latent state and separate positional components. Sliding-window attention only needs a recent region. Recurrent and state-space layers can keep fixed-size state instead of one entry per token.

This means that a modern engine may manage several kinds of persistent state in the same model. Calling all of it “the KV cache” is convenient, but assuming it has one shape or one retention rule is not.

Attention patterns change what can be reused

Full causal attention allows a new token to attend to every earlier token. Other patterns limit the receptive field.

A sliding-window layer attends only to recent positions. A local or block-sparse layer follows a fixed pattern. Cross-attention reads state produced by an encoder. Some architectures share state between layers or summarize old positions into recurrent state.

Each pattern changes what persistence means. A sliding-window layer’s old entries eventually become dead weight that a clever engine could release; a full-causal layer’s entries remain load-bearing until the sequence ends. Cross-attention state belongs to the encoded input rather than the generated text, so it can be reused across questions about the same document. Reuse rules are pattern rules, decided by the architecture and discovered by the engine.

The table is an eviction-policy decision table in disguise: what a cache manager may release, and when, follows from the row the model occupies. A policy tuned for full causality hoards sliding-window dead weight; a policy that frees aggressively breaks cross-attention reuse.

The model runner must pass the correct positions, mask, page table, and layer-specific metadata to the attention implementation. Selecting an attention backend is therefore a correctness decision before it becomes a performance decision: a backend that assumes full causality on a sliding-window model produces plausible-looking output with silently wrong attention, a failure mode worse than crashing because nothing reports it. Chapter 8 treats backend selection as a match against device, dtype, cache layout, and execution mode together.

Mixture-of-experts models add routing

A dense feed-forward layer applies the same parameters to every token. A mixture-of-experts, or MoE, layer contains many feed-forward networks called experts. A router selects a small number of experts for each token.

This conditional computation allows the model to contain many parameters without using all of them for every token. The accounting distinction that matters for serving is resident versus active: a model with eight 7-billion-parameter experts keeps roughly 56 billion parameters’ worth of expert weights in memory, yet applies only about 14 billion per token when two experts are selected. Memory planning must satisfy the resident number; compute planning scales with the active one. The gap between them is where MoE serving gets interesting.

It also introduces irregular work. Tokens must be grouped by expert so the accelerator can run efficient matrix multiplications. If experts live on different GPUs, token representations must move to the selected owners and return afterward.

The busiest expert determines when the step finishes. A model with balanced average routing can still have a hot expert for a particular workload. Serving MoE models is therefore as much a placement and communication problem as a matrix-multiplication problem, and Chapters 12 and 13 give it dedicated treatment.

Watching one batch route

Follow sixteen decode tokens through one MoE layer with eight experts, top-2 routing. The router scores each token against all eight experts and picks the two highest. Counting assignments across the sixteen tokens might yield loads of 7, 5, 5, 4, 4, 3, 2, 2 — thirty-two assignments, sixteen tokens, two each, yet nothing like uniform. The engine groups tokens by chosen expert so each expert runs one batched matrix multiply instead of sixteen small ones; expert 0 processes seven tokens while expert 7 processes two.

If the experts live on different devices, each token’s hidden state travels to its two selected owners and the weighted outputs travel back — an all-to-all exchange whose volume is set by routing decisions made milliseconds earlier. The step cannot finish until expert 0 finishes, so stragglers set the pace: the lighter-loaded devices wait. Averaged over a whole workload the router may look balanced while individual steps swing widely, which is why MoE schedulers reason about per-step loads rather than long-run averages. What balancing, placement, and capacity policies do about it is the business of Chapters 12 and 13; the point here is that the irregularity is created by the model’s own forward pass, not by the server.

Sampling has state too

The model’s logits are not always sampled directly. Temperature, top-k and top-p filters, repetition penalties, token bans, grammars, and custom logit processors can all change the distribution. These processors form a chain applied in a defined order, and the order is part of behavior: penalizing repetition before or after top-k filtering yields different outputs from the same logits. Random sampling owns a generator state — seed and stream position — that belongs to the request and must survive across steps. Structured generation owns a parser or finite-state machine for each sequence, updated as tokens are emitted.

Greedy selection chooses the highest-scoring token. It is deterministic only if the logits and tie handling are identical. Different batch shapes, reduction orders, kernels, precisions, collectives, or cache paths can slightly change the logits. Temperature zero does not by itself guarantee identical output across executions. Later chapters will separate deterministic selection from deterministic numerical execution — the first lives in the sampler, the second in kernels and compilation, and confusing them produces debugging sessions that search the wrong layer.

One distribution through the processor chain

A tiny vocabulary makes the chain inspectable. Suppose a step’s logits give five candidate tokens these probabilities: A 0.60, B 0.25, C 0.10, D 0.04, E 0.01. Temperature first: dividing the logits by 0.5 and re-normalizing sharpens the distribution — A rises well above 0.60, the tail flattens toward zero. Dividing by 2.0 instead flattens it — A falls, the tail gains mass. Temperature never reorders tokens; it changes how much probability mass the ordering carries.

Now top-k with k = 2: keep A and B, zero the rest, renormalize. C, D, and E become unreachable this step no matter how the dice fall. Top-p with p = 0.90 instead keeps the smallest set whose cumulative mass reaches 0.90 — A plus B plus C reaches 0.95, so the candidate set is those three, and implementations genuinely differ on whether C, the token that crosses the threshold, is kept or dropped. That ambiguity is exactly why processor order and semantics are part of a service’s contract: repetition-penalizing A before the top-k cut can push B into the surviving set; applying the same penalty after the cut changes nothing, because B’s fate was already decided. Two servers can expose identical parameter names and produce different distributions because their chains apply the same operations in a different order — a compatibility hazard Chapter 5 returns to when it defines what an execution request must carry.

Models with encoders

Now consider a user who attaches an image to a question. The service may decode the image, resize and normalize it, run a vision encoder, project the resulting features, and insert them into the language-model input. Only then do language prefill and decode begin.

image bytes -> preprocessing -> vision encoder -> media features
                                                    |
text ----------> tokenization ----------------------+
                                                    v
                                      language prefill -> decode

For a large image or video, the encoder can dominate time to first token: a high-resolution image or a minute of video can produce thousands of feature tokens, each of which then occupies positions in the language model’s context. The token count follows from geometry. A vision encoder typically splits the image into fixed-size patches — a 448 × 448 image cut into 14-pixel patches yields 32 × 32 = 1,024 of them. Double the resolution and the patch count quadruples; add frames to a video and the counts multiply again. This is why a service can budget carefully for text length and still be ambushed by media: one resolution setting change moves the language model’s context cost by multiples, and the practice exercise’s 2,048 feature tokens per image is exactly this arithmetic at work.

Encoder output may be reusable when the user asks several questions about the same media — the features, unlike the question, do not change. Resolution and frame count also create dynamic shapes: two requests differ in encoder workload even when their text is identical length. The encoder is a serving stage with its own batching, caching, and placement decisions—not a minor preprocessing detail, and Chapter 18 promotes it to a first-class workload.

Embedding, reranking, classification, and reward models often have no decode loop at all. They batch complete inputs and produce complete outputs. Engines that support these tasks need schedules and output paths suited to them — there is no stream to pace and no state to extend, but there is also no partial progress to show a waiting caller.

Diffusion follows a different loop

An image-generation pipeline commonly contains a text encoder, a denoising network, a scheduler that chooses noise levels, and a decoder that turns a latent representation into pixels. The denoising network runs many times — often twenty to fifty denoising steps — with each pass refining the same latent.

Unlike an autoregressive sequence, a diffusion request often advances all spatial positions together. Its latent state may keep a stable shape across steps, which makes its per-step work predictable in a way decode is not. Requests can share a batch when their resolution, step, conditioning, and backend requirements are compatible. Some systems cache repeated work between nearby denoising steps, exploiting the fact that consecutive passes change the latent only slightly.

The lesson is broader than diffusion: an inference engine should model stages, dependencies, and state rather than assume that every request emits one token per step. An engine designed around the decoder loop alone will force every other topology through shapes that fit it badly.

From model topology to serving topology

The model topology describes what depends on what: layers, experts, encoders, attention state, and iterative stages. The serving topology maps that work onto devices and processes.

Different model families create different serving graphs.

flowchart TB
    R["Request"] --> T{"Model topology"}
    T --> A["Dense decoder: repeated token loop"]
    T --> B["MoE decoder: route to experts"]
    T --> C["Multimodal: encoder then decoder"]
    T --> D["Diffusion: repeated denoising loop"]

The first diagram carries the chapter’s central dependency: the arrow from the selected token back into both the state and the next step. Everything the scheduler struggles with — serialization, preemption costs, speculative execution — traces back to that feedback edge. The second diagram is a dispatch table: the request’s topology decides which serving graph it enters, and each branch carries its own persistent state and irregular work.

The same model can be replicated in full, split across devices by tensor or layer, distributed by expert, or separated into encoder, prefill, and decode pools. All may be legal. The workload, hardware links, memory capacity, and SLO decide which is useful. Reading the model topology tells an engineer which splits are even available: a model whose experts dominate its parameters has an expert-parallel option a dense model lacks; a model with a heavy encoder has an encoder-disaggregation option a text-only model lacks.

The rest of the book applies this inventory repeatedly. The immediate next step is the hardware those topologies must live on.

Worked example: inventory a dense decoder

Consider a BF16 dense decoder with 70 billion parameters, 80 layers, 8 KV heads, and head dimension 128. Its parameter storage is roughly 140 GB before runtime overhead — 70 billion parameters at 2 bytes each. Using the cache formula from this chapter, each token creates 320 KiB of KV state across the model, as the applied calculation above showed. An 8,000-token sequence therefore needs about 2.44 GiB.

Those two numbers immediately constrain serving. The weights do not fit on one 80-GiB device at BF16, so some parallel split is mandatory before the first request arrives — four ways, say, leaving about 35 GB of weights per device. Long active contexts consume memory on top of that: each device also owes its quarter of every sequence’s state, roughly 625 MiB for one 8,000-token sequence, and a batch holds many sequences. Prefill creates many token positions at once; decode repeatedly reads the sharded weights and existing state for one new position per sequence.

The same inventory for an MoE model must separate total resident expert weights from experts active per token — the 56-versus-14 billion distinction above. For a vision-language model it must separate the encoder, projected media features, and decoder KV state. Different inventories lead to different legal placement plans.

Practice: compare three model topologies

Inventory the dense decoder above, an 8 × 7B expert model with two experts active per token, and a vision-language model that produces 2,048 feature tokens per image. For each, list parameter bytes, persistent state, prefill and decode shapes, conditional communication, and independently placeable stages.

Do not choose an engine setting yet. Produce the facts a serving plan must respect. The worked inventory is in Appendix G.

4. Hardware Is a Topology

A machine specification says that a server has eight GPUs. That sounds precise, but it leaves out the information an inference engineer needs most.

Can every GPU communicate directly with every other GPU? Do four devices sit behind one CPU socket and four behind another? Is there one network interface or several? Which links are shared? Eight identical accelerators can form very different systems.

The omitted facts decide real outcomes. A tensor-parallel group striped across two islands runs the same arithmetic measurably slower than one kept inside a fast island. A tokenizer pool pinned to the wrong socket steals bandwidth from a network progress thread. Two replicas that look independent fail together because they share a switch. None of these appear in a specification sheet; all of them appear in production incidents.

Hardware is not a bag of processors. It is a map of places where bytes can live and paths along which bytes can move.

Four limits to keep separate

Hardware discussions often collapse everything into “speed.” In practice, four resources matter.

Compute rate describes how many arithmetic operations a device can perform per second. Capacity describes how many bytes fit in a memory tier. Bandwidth describes how quickly bytes move once a transfer is underway. Latency describes how long a dependency takes, including the cost of starting it.

A device can have enormous compute rate and still wait on memory. A network can have high peak bandwidth but perform poorly for the tiny, frequent messages of decode. A model can fit in device memory and still run out of space when the engine reserves KV blocks, graph workspaces, and collective buffers.

Keep the four limits separate until measurements show which one governs the workload.

Four limits, one decode step

One decode step from Chapter 3’s dense decoder touches all four limits, at different stages. Walk the step and ask what could bind at each point.

Starting the kernels costs microseconds of launch latency each — pure dependency overhead, paid even though no meaningful arithmetic or traffic has happened yet; at small batch with short steps, these microseconds are a real fraction of the step. Inside the attention kernel, the arithmetic itself is trivial compared with streaming 140 GB of weights and each sequence’s accumulated state through the memory system, so bandwidth binds while the compute units idle. If the model is sharded four ways, each layer ends with a reduction whose duration depends on fabric latency and on the slowest rank — a synchronization limit that no amount of local bandwidth fixes. And before any of this, admission had to find room for the sequence’s state: a capacity limit that decides whether the step runs at all.

The practical consequence is diagnostic. A slow step could be any of the four, and each has a different fix — fewer launches, fewer bytes, better placement, or stricter admission. Treating “the GPU is slow” as one problem produces optimizations aimed at the wrong limit; Chapter 8’s profiling discipline exists largely to tell them apart.

A practical roofline

Arithmetic intensity measures how much computation an operation performs for each byte it moves from a chosen memory tier:

The roofline question chooses the first optimization direction.

flowchart TB
    O["Measure operation and byte traffic"] --> I["Compute arithmetic intensity"]
    I --> X{"Below compute-to-bandwidth crossover?"}
    X -->|Yes| B["Reduce bytes or improve locality"]
    X -->|No| C["Reduce arithmetic or use faster compute"]
    B --> V["Verify end-to-end bottleneck"]
    C --> V

The first diagram is a cost ladder: each hop outward buys capacity with latency and bandwidth, and the optimizer’s job is to keep hot objects on the lowest rung their access pattern justifies. The second diagram is a triage procedure, not a description of the machine — it produces a hypothesis about which resource governs, and the final node insists that measurement confirm or falsify it before anyone optimizes.

arithmetic intensity = operations / bytes transferred

If a device can perform C operations per second and the relevant memory path can move B bytes per second, a simple upper bound is:

attainable rate <= min(C, B * arithmetic intensity)

Below the crossover point C / B, the operation is limited by moving data. Above it, compute becomes the tighter ceiling. This roofline model is simple, but it asks the right first question: should you reduce arithmetic, reduce traffic, or improve overlap?

Apply the model at the boundary that matters. A decode layer may be limited by GPU memory bandwidth while the entire model step waits on a cross-node collective. A cache load may be fast from local storage and slow across PCIe. There is not one roofline for the whole service.

Where the crossover sits for decode

Make the crossover concrete with declared assumptions. Suppose an accelerator performs about one quadrillion arithmetic operations per second and its memory system moves about three trillion bytes per second. The crossover intensity is then roughly 333 operations per byte: work below that intensity is bandwidth-bound no matter how idle the arithmetic units look.

Now place decode on that scale. Reading BF16 weights costs 2 bytes per parameter, and each parameter contributes about two operations per sequence in the batch — so the intensity of the weight read, in operations per byte, is approximately the batch size. Batch 8 sits at intensity 8, more than a factor of forty below the crossover: utterly bandwidth-bound. Batch 128 approaches intensity 128 and starts to matter computationally. The exact crossover varies by orders of magnitude across hardware generations, but the shape of the conclusion does not: small-batch decode lives far below the roofline’s knee, which is why Chapter 3 called it a streaming workload, and why the scheduler’s batch composition is a hardware-efficiency decision as much as a latency one.

Naming the ceiling: utilization and MFU

The field compresses this roofline reasoning into one number: model FLOPs utilization (MFU) — achieved useful arithmetic divided by the accelerator’s peak, where “useful” counts the two operations per parameter that the math requires and ignores everything the hardware wastes on data movement, padding, or unsupported shapes. The vocabulary earns its keep because it makes expectations quantitative before any measurement. Prefill is large, dense matrix work executed largely on tensor cores; declared planning ranges for well-configured serving put it in the tens of percent — call it 30 to 60. Decode at batch 1 has intensity near 1 against a crossover in the hundreds, so its MFU ceiling is roughly batch / crossover — for the numbers above, under half a percent at batch 1, a few percent by batch 16. When someone reports decode MFU of 40 percent, they have either measured prefill, misused the term, or built something extraordinary; asking which is a better question than believing the number.

Two disciplines follow. First, quote MFU only with its regime: prefill-MFU and decode-MFU are different quantities with different ceilings, and neither is comparable across batch sizes without the intensity context. Second, use the arithmetic to set expectations before benchmarking: if a planned configuration implies 60 percent decode MFU at batch 4, the plan contradicts arithmetic, not tuning. Chapter 8 returns to the tile level — where tensor cores consume whole rectangles of data and explain why the crossover exists — and Chapter 23 turns these ceilings into full capacity estimates.

Prefill tells the opposite story. A 1,000-token prompt performs a thousand positions’ arithmetic per weight read, putting its intensity near a thousand — comfortably above typical crossovers. The same weights, the same device, and the binding limit flips. Any mechanism that mixes phases inherits both profiles in one schedule, which is exactly why phase-mixing is hard.

The crossover also sorts the book’s remaining mechanisms into two families. Below it, the winning moves reduce bytes or improve reuse: paged allocation (Chapter 7) stops state fragmentation from wasting capacity, quantization (Chapter 10) shrinks the weight bytes themselves, and batching raises intensity directly. Above it, bytes are no longer the constraint, so the winning moves cut arithmetic or overlap it: speculative decoding (Chapter 11) spends extra arithmetic to shorten the critical path, and disaggregation (Chapter 15) moves work to where its limiting resource is plentiful. When a profiler says a kernel sits below the crossover, that ordering tells you which chapter to open first.

Why batching changes hardware efficiency

During a long prefill, large matrix operations reuse model weights across many token positions. That gives the GPU substantial work for each byte of weights it reads.

During decode, a small batch may read nearly the same weights to process only a few new positions. The operation is more likely to be memory-bound. Adding sequences to the batch allows one weight read to support more useful work.

This is why larger batches often improve throughput. It is also why throughput and latency conflict: requests may need to wait until enough compatible work is available, and a larger step itself takes longer. The scheduler chooses where the service operates on this curve.

The waiting term is not mysterious. If requests arrive at twenty per second and the scheduler wants eight per batch, collecting them costs on average 8/20 = 0.4 seconds of added time to first token — simple arithmetic, but the kind that decides SLOs. Waiting longer buys hardware efficiency at a rate the arrival process sets, which is why continuous batching (Chapter 6) refuses to wait for full batches and instead admits whatever is compatible each step: it keeps most of the efficiency gain while charging almost none of the queueing cost.

Where the bytes live

An inference deployment may use a hierarchy that begins with registers and on-chip scratch memory, continues through device caches and high-bandwidth memory, and extends to host memory, local storage, remote storage, and durable object storage.

Every byte follows a physical path, even when the API hides it.

flowchart LR
    R["GPU registers and SRAM"] --> H["Device HBM"]
    H --> P["PCIe or local GPU fabric"]
    P --> M["Host memory and NUMA socket"]
    M --> N["NIC and network fabric"]
    N --> S["Remote memory or storage"]

Closer tiers are scarce and fast. Farther tiers provide more capacity at higher access cost. Different objects deserve different treatment. Model weights are large and repeatedly read. KV blocks grow with active sequences and may be reused. Compiled graphs are expensive to recreate but tied to an execution environment. Adapters and media embeddings have their own popularity patterns. An object’s right tier can also change over its lifetime: sequence state is hot while the request runs, cold the moment it is preempted, and dead at completion — three different storage problems wearing one name.

When checking whether a model fits, include the whole working set:

weights
+ persistent request state
+ temporary activations
+ communication buffers
+ graph and compiler memory pools
+ allocator headroom

Parameter size alone is not a capacity plan.

Turning the fit equation into an admission budget

Walk the fit list with Chapter 3’s decoder on the worked deployment below: an 80-GiB device holding one rank of a four-way split. Weights take about 35 GB — and note the trap: gigabytes are not gibibytes, and 35 GB is roughly 33 GiB, so the honest remainder is about 47 GiB, not 45.

Reserve, say, 12 GiB for activations, communication buffers, graph pools, and allocator headroom — a declared planning assumption, not a measurement. That leaves about 35 GiB for persistent state. Each 8,000-token sequence owes its rank one quarter of 2.44 GiB, roughly 0.61 GiB, so the device admits around fifty-seven such sequences. Fifty-seven is now an admission number: a scheduler accepting the fifty-eighth long conversation without evicting another is promising memory the device does not have, and Chapter 7’s paged allocation exists to make that accounting exact rather than approximate.

The same walk explains a common production surprise. The deployment ships, fits comfortably, and serves short conversations for weeks. Users discover long-document chat, contexts drift toward the maximum, and the KV share of the working set quietly triples. Nothing was misconfigured; the fit assumptions were, because capacity planning used the context lengths of the pilot, not the ones the workload grew into.

Which tier for which object

The fit equation says what must fit; the tier ladder says where each object should live. The decision follows each object’s access pattern, and the four main objects disagree with each other.

Weights are read in full every step and never change within a deployment, so they belong on the highest tier that fits — anything farther costs bandwidth on every single step. KV state is the opposite: append-heavy while a sequence lives, dead the moment it ends, and the only major object whose total size the scheduler controls by admitting or evicting sequences. That controllability is what makes offloading it plausible at all. Compiled graphs are read constantly but are small next to weights and are invalidated by environment changes, so device or host memory suits them. Media embeddings are reused across requests about the same input, so they want a tier near the engine with an eviction policy — a cache, not a residency guarantee.

The cost gap between tiers is the whole argument. Assume, as declared planning numbers, a device memory system that moves about three trillion bytes per second and a device-to-host path that manages about fifty billion — a sixty-fold difference. One rank’s share of an 8,000-token sequence, 0.61 GiB, streams from device memory in well under a millisecond but takes on the order of thirteen milliseconds to pull back from host memory. A preempted sequence whose state was swapped to the host does not resume for free; it replays a ten-millisecond-scale penalty into some unlucky request’s inter-token latency — a visible bite out of a 150-millisecond budget. That is why preemption policy (Chapter 6) and KV transfer design (Chapter 15) treat tier placement as a latency decision, not a storage detail.

Links decide which parallel plans make sense

Within a host, devices may communicate over PCIe or a higher-bandwidth GPU fabric. Across hosts, data may travel over RDMA-capable networks. Exact product names change, but the design questions stay stable.

Which pairs have direct peer access? Which transfers cross a CPU root complex? Which ranks share a switch or network rail? Can a transfer overlap the kernels that surround it? What happens when every rank communicates at once?

Topology inventory is evidence, not a performance result. NVIDIA’s DCGM topology guide explicitly separates known CPU, PCIe, and NVLink relationships from active path tests and observed traffic. Use the analogous inventory and diagnostic tools for the platform being measured.

Different forms of model parallelism create different traffic:

Tensor parallelism tolerates slow links worst: its collectives sit inside the critical path at layer frequency, so link quality multiplies across eighty layers. Pipeline parallelism hides link latency better but pays in bubbles. Disaggregation moves few, large messages, which favors paths chosen for bandwidth over latency. Matching each pattern’s traffic shape to the links that suit it is the placement problem Chapters 10 and 14 solve concretely.

The important quantities are message size, frequency, synchronization, and path—not only total bytes.

The last design question — what happens when every rank communicates at once — deserves its own attention, because aggregate bandwidth is rarely the sum of per-link bandwidth. Eight ranks that all reduce through one switch contend for its backplane; a dual-rail design where each rank owns one rail carries two full-width collectives concurrently, while a design that pins half the ranks to each rail but lets collectives span both pays a bridging hop on every message. Stragglers amplify the contention: a collective ends when its last participant arrives, so one rank whose path is oversubscribed stretches all eight. These are placement outcomes — visible in a topology drawing, invisible in a per-link bandwidth specification.

CPUs remain on the critical path

The GPU runs the model, but the CPU may parse requests, tokenize text, preprocess media, make schedules, prepare metadata, coordinate transfers, and turn outputs into stream events. Once GPU steps become short, Python work or a host synchronization can take a large fraction of each engine step.

A declared-assumption arithmetic makes the exposure vivid. Suppose an engine step takes six milliseconds on the device and the host needs four more per step — sampling bookkeeping, metadata assembly, stream polling — serialized before the next launch. The device then runs at most sixty percent duty cycle no matter how its kernels are tuned, and the missing forty percent will never appear in a GPU profile as memory or compute time. This is why Chapter 1’s engine separates launch path from scheduler path, and why captured graphs and overlapping event loops exist: they attack the host-side term, not the device-side one.

CPU placement also matters. A process can access memory attached to another NUMA node or control a device behind another CPU socket. Tokenizer pools can compete with network progress threads. Unified addressing can make memory accessible without making it local or fast.

The penalty has a shape worth internalizing: it is per-interaction, and inference is dense with interactions. A scheduler process on socket B driving a device attached to socket A pays the crossing cost on every doorbell, metadata write, and completion poll — individually small, but multiplied by thousands of interactions per second, the crossings become a measurable fraction of the host budget from the six-millisecond-step arithmetic above. Pinning the scheduler, its tokenizer pool, and its network progress threads to the sockets that own their devices removes the crossings without making any single interaction faster — a placement fix, not a code fix, which is why it belongs in the topology drawing rather than the profiler’s hot path.

Measure CPU time, run-queue delay, memory placement, and synchronization beside GPU utilization. A GPU that appears underused may be waiting for the host.

Draw the physical map

Suppose you have 16 GPUs arranged as two groups of eight with fast links inside each group and a slower network between them. A tensor-parallel group of eight should usually fit within one fast island. Alternating its ranks across both islands changes no arithmetic, but forces frequent collectives onto the slower path.

Your topology drawing should follow each rank all the way out:

rank -> accelerator -> local fabric -> CPU socket -> NIC -> switch -> rack

Now add traffic. Mark how many bytes cross each logical edge and how often. A topology diagram without traffic is an inventory; adding traffic turns it into a performance hypothesis.

Annotating the four-replica deployment shows how uneven the traffic lands. Every intra-replica edge carries two reductions per layer, per step — with eighty layers and dozens of steps per second, thousands of small messages per second on the island fabric, each on the critical path of every rank in the group. The inter-island edge carries almost nothing in steady state — no collective, no activation — until a KV handoff or a cache hit crosses it, and then it moves megabytes in one burst. Optimizing the busy edge means lower latency; optimizing the quiet edge means higher burst bandwidth. The annotated drawing makes it impossible to spend effort on the wrong one.

Topology also defines failure. If one request needs every rank in a parallel group, losing one rank can stop the group. Two replicas placed behind the same power or network boundary do not provide the independence their count suggests. A remote cache can improve warm-start performance and become a shared failure dependency at the same time.

Work the two-island scenario to its failure conclusion. The natural placement puts one four-rank replica per island — but if both replicas’ network interfaces hang off the same switch, that single switch is now a capacity-zero event for the whole service: one device fails, both replicas become unreachable, and the replica count of two turns out to have been an availability claim the physical map never supported. Splitting the replicas’ paths across switches or rails restores genuine independence without changing any model arithmetic. The general rule: independence is a property of the physical map, not of the replica count, and every shared edge in the drawing — power, switch, cache, storage — is a correlated-failure candidate.

Worked example: place before measuring

Place the dense model from Chapter 3 on two eight-GPU nodes with 80 GiB per GPU. Fast links connect devices inside each node; the inter-node path is slower. A four-way tensor-parallel replica holds about 35 GB of weights per rank before overhead, and each rank holds roughly one quarter of the KV state.

Keep every four-rank group inside one fast-link island. Striping alternating ranks across nodes changes no model arithmetic but moves layer-frequency collectives onto the slower network. That is a topology error visible before a profiler runs.

Predict long prefill to stress compute and attention traffic, batch-1 decode to stress device-memory bandwidth and collective latency, and a 2.44-GiB KV move to follow the slowest staging or network edge. The profiler’s job is to falsify those claims. Large CPU gaps during decode, for example, would reveal a host or launch bottleneck the prediction omitted.

Practice: write a falsifiable hardware prediction

Draw both nodes through GPU links, CPU sockets, NICs, and the connecting switch. Place two four-way replicas, calculate per-rank weight and 8,000-token KV bytes, and mark every collective and state-transfer path.

Predict the limiting resource for long prefill, batch-1 decode, and remote KV load. For each prediction, name a counter or timeline observation that would prove it wrong. Compare with Appendix G.

That habit—predict, measure, explain—will be more useful than memorizing any hardware table. Part II now follows a request through the software that turns this topology into an executing service.

Part II — Inside a Single Engine

One request, one engine, opened end to end. A scheduler chooses the next unit of work, a memory manager finds room for its state, kernels move the bytes, compiled graphs remove the launch cost, and the numerical and decoding choices determine what the engine may safely trade away. Chapter 12 extends the single-engine story to multi-tenant adapter serving, where many customized models share one set of base weights.

Chapters 5–12

5. Anatomy of an Inference Server

In the previous chapters, we described the work, the model, and the hardware. Now we can follow a request through the software that connects them.

Assume a user sends this chat request:

System: You answer questions about Acme products.
User: Why is my device blinking amber?

The request looks simple. By the time its first token reaches the user, several components have made decisions on its behalf. An API process decided whether the request was well-formed before any expensive resource was touched. A scheduler decided when it would run and what would share its step. A worker decided which kernels and graph shapes would execute it. An output processor decided what the user’s stream would actually show — and when to throw a computed result away.

Each of those decisions lives at a boundary. The chapter’s job is to name the boundaries precisely enough that you can reason about their failures: a request accepted twice, a block table pointing at freed pages, a token delivered after cancellation. Servers rarely fail in the middle of a component; they fail in the contracts between them.

The public request becomes an internal request

The frontend receives the HTTP or RPC message. It authenticates the caller, checks the requested model, validates generation parameters, and applies limits on input size and output length.

The engine separates user-facing work from step-critical execution.

flowchart LR
    A["API process"] --> B["Input and tokenization"]
    B --> C["Engine core"]
    C --> D["Scheduler"]
    D --> E["Executor"]
    E --> F["Worker and model runner"]
    F --> G["Output processor"]
    G --> A

For a chat endpoint, the messages are not yet the model input. A chat template turns roles and content into formatted text. A tokenizer converts that text to token IDs. Tool definitions, images, audio, adapters, and structured-output rules may add more processing.

At the end of this stage, the engine needs an internal request that describes exactly what will execute. That distinction is easy to miss:

• The API request records what the caller sent.
• The execution request records the resulting tokens, processors, model version, adapter, media features, positions, sampling rules, and deadline.

Cache identity must follow the execution request. Two equal prompt strings can produce different tokens after a template or tokenizer change. Reusing state because the strings match would be incorrect.

Validation should also finish before expensive state is reserved. A service should reject an unsupported parameter or an oversized image before it occupies GPU memory. Compressed media deserves special care because a small request body can expand into a large decoded tensor. A 2 MB upload is not a 2 MB cost; the question an admission check must answer is how many pixels, patches, or feature tokens it becomes — Chapter 3’s patch-count arithmetic, applied before the encoder ever runs.

The ordering of these checks is itself a small design. Cheap, decisive checks run first: authentication, model existence, parameter ranges, size limits. Tokenization follows because it can fail (unknown characters, length overflow) and because everything downstream consumes its output. Media decoding runs last of the expensive steps, since it is the one most likely to reveal a request that should never have been admitted — and by then the only honest outcome is a fast, explicit rejection. An admission path that decodes media before validating token limits has inverted the order and will pay for the inversion at the worst time: under load, when rejected work is most expensive.

The request enters the engine

Once prepared, the request enters a waiting queue. It does not immediately become a GPU batch. The scheduler first considers the work already running, available memory, the step’s token budget, request priority, and any reusable prefix state.

The scheduler’s output is a plan for one engine step. It might say:

Process 512 prompt tokens for request A, produce one decode token for requests B through K, use these KV blocks, and release the state for request J after its output is consumed.

The exact representation varies, but the idea is stable. A schedule connects a policy decision on the CPU to concrete input preparation on one or more workers.

The scheduler works closely with a state allocator. For a language model, the allocator maps logical token positions to physical KV-cache blocks. For a multimodal request, another cache may own encoder outputs. Allocation must succeed on every required rank before the schedule is safe to execute — a partial allocation across ranks is worse than none, because the schedule that references it cannot be rolled back cheaply once any rank has begun executing.

What the block table has to get right

The allocator’s contract with the runner deserves a close look, because attention correctness hangs on it. The block table is the mapping from a sequence’s logical positions to the physical pages that hold its state; the runner trusts it completely, and the kernel indexes memory through it. Three events stress the contract.

A prefix-cache hit hands a request state it did not compute: the table now points at blocks shared with other sequences, which must be treated as read-only until this sequence appends past them — at which point the shared tail needs copy-on-write semantics, or the next append corrupts a stranger’s context. A preemption revokes the table: the sequence’s pages return to the pool or move to a slower tier, and any schedule still holding the old table is stale by definition — this is where the version discipline from the output path reappears on the input side. A finish releases pages that a cache may want to retain as a reusable prefix, so “release” splits into two decisions: free for allocation, and retain for reuse, with different lifetimes.

Each event has a silent failure mode: corrupted shared tails look like model quality regressions, stale tables look like crashes or garbage tokens, and premature release looks like a cache that never hits. Chapter 7 builds the data structures that make these events cheap; the point here is that the scheduler-to-worker boundary carries a live contract, not just a data structure.

Executors, workers, and model runners

The next layers are often confused because a small server can combine them in one process.

An executor decides which workers participate in an engine operation and how to communicate with them. A worker owns the resources for one device or rank: device context, distributed groups, memory pools, and loaded model. A model runner turns the schedule into tensors and invokes the model, kernels, graphs, and collectives for that device.

The separation becomes useful on multiple GPUs. The executor knows that eight ranks must run. Each worker knows its local shard and communication groups. The model runner knows which graph shape and attention metadata are needed for this step.

Combining these roles can reduce messages and process overhead. Separating them can isolate failures and support several execution backends. Neither layout is automatically better; you should judge it by ownership, synchronization, and failure behavior.

One step, three processes

Make the separation concrete by following the Acme request’s first decode step through a fully split deployment. The API process holds the HTTP connection and the user’s stream; it has already tokenized and validated, so it sends a compact execution request across a process boundary. The engine-core process runs the scheduler and allocator: it decides the step’s membership, assigns blocks, and emits a step plan toward the workers. Each worker process — four of them for a four-way split — prepares local tensors from the plan, runs its shard, participates in the layer collectives, and reports sampled results back toward the output path.

Three hops, each with its own serialization and queue. The hops cost microseconds each — negligible against a multi-millisecond engine step, and still small against a TTFT budget measured in hundreds of milliseconds. What the separation buys is not speed but isolation: the API server’s event loop, with its slow client connections and Python detokenization, cannot starve the scheduler’s loop; a worker crash is visible as a dead rank rather than a dead server; and the scheduler can be restarted or upgraded independently of the processes holding GPU memory. The cost is that every one of those hops needs the ownership and failure answers the previous section demanded — which is why the source systems in this chapter put real machinery, not just function calls, at each boundary.

The Acme request from the chapter’s opening crossed these boundaries twice: downward as an execution request and then a step plan, upward as sampled tokens and stream events. Every decision this chapter named happened in between, and each one left a trace you now know how to look for.

From logits back to a stream

The model runner returns logits or another task-specific result. For text generation, the sampler applies temperature, top-k or top-p rules, penalties, random state, and output constraints. It selects the next token ID.

Overlapped output work needs versions to discard stale results.

flowchart TB
    A["Step N completes on device"] --> B["Results queued for output processing"]
    A --> C["Step N+1 launches without waiting"]
    B --> D{"Request still active under this version?"}
    C --> D
    D -->|Yes| E["Apply result and advance state"]
    D -->|No| F["Discard as stale"]

The first two diagrams divide the server along its stable seam: user-facing latency work runs ahead of execution, step-critical work runs behind it, and the boundary table below says what must be true at each crossing. The third diagram is the price of that separation. Once output processing lags execution, results can arrive for requests whose state has already moved on, and only a version discipline turns a corruption bug into a discarded message.

Output processing then updates the request. It checks stop conditions, advances a grammar or tool parser, converts token IDs to text, updates usage counters, and creates streaming events. When the request finishes or is cancelled, it also arranges for state to be released or retained as a reusable prefix.

This work happens on every decode step. If the GPU must wait for Python detokenization and network serialization before it can begin the next step, output processing becomes part of the critical path.

Engines often overlap output work with the next GPU operation. The price of that overlap is bookkeeping. A preempted or cancelled request may have an old result still in flight. The result needs a step or state version so the engine can recognize and discard it — the third diagram’s decision, implemented rather than admired. Stop-string detection adds a subtlety: a stop string may only become visible after several tokens have been processed together, so the engine may learn of a finish condition after steps beyond the true stop point have already executed, and must abort those follow-on steps deliberately.

Where sampling runs, and why it matters

The sampler’s placement is a real architecture decision, and the logits make the stakes countable. Chapter 3’s decoder emits 128,000 scores per sequence; at four bytes each, one sequence’s logits are about 512 KB. Copying them from device to host every step for CPU-side sampling costs half a megabyte per sequence per step — at a batch of sixty-four, 32 MB per step against a memory system that Chapter 4 valued in trillions of bytes per second but a PCIe budget measured in tens of billions. The copy is pure overhead: the device that just produced the scores is also the cheapest place to filter and select from them.

Placement interacts with structured output. A grammar or JSON-schema constraint must mask forbidden logits before selection, and the mask state — the parser’s current position — usually lives host-side with the request. So each step carries a small host-to-device journey for the mask and a small device-to-host journey for the chosen token, and engines work hard to keep both off the critical path, batching mask construction or computing it on device. The design lesson generalizes: state lives where it is updated, and the sampler’s state is updated every step — which is why Chapter 3 called sampling stateful, and why this chapter’s boundary table gives the worker-to-output hop its own invariant row.

Messages are not all alike

An inference server carries several kinds of traffic between its components. Schedules and lifecycle commands are small control messages. Tokens, positions, and block tables are metadata. Streamed outputs and metrics flow back toward the frontend. KV blocks and encoder embeddings are bulk data.

Control messages and bulk data take related but distinct paths.

flowchart TB
    R["Request record"] --> S["Schedule metadata"]
    S --> W["Worker command"]
    W --> C["Completion event"]
    T["Token tensors"] --> M["Model execution"]
    K["KV blocks"] <--> M
    M --> L["Logits and sampled IDs"]

Using the same channel for all four creates problems. A large state transfer can delay a cancellation command. A serialization format designed for convenient objects can waste CPU on every decode step. A local queue can hide the absence of backpressure once workers move across a network.

Representation choice is where that CPU goes. Suppose a batch of sixty-four sequences at 8,000 positions each needs its block tables delivered every step. As paged metadata — sixty-four tables over, say, sixteen-token pages — that is a few hundred integers per sequence and kilobytes overall. As a naive per-token list in a general-purpose text format, it is half a million positions serialized as individual values: megabytes parsed on the host, every step, to describe memory the device already holds. Same information, three orders of magnitude apart, purely a representation decision — and the step plan crosses this boundary at engine-step frequency, which is why the boundary table demands “metadata matches allocation” rather than “metadata is complete.”

For each channel, document ordering, serialization, ownership, backpressure, and failure. If a sender dies after transferring data but before acknowledging it, who owns the buffer? If the receiver restarts, can it distinguish a delayed message from current work? These questions sound bureaucratic until the first time a restart produces duplicated tokens in a paid stream; then they become the checklist you wish you had written.

Where the channels actually run

The four traffic classes end up on physically different transports in a mature deployment, and the reasons are worth tracing. Control messages ride a small local socket or IPC queue: tiny payloads, strict ordering, and the receiver must never be busy long enough to delay a cancellation behind bulk work — head-of-line blocking here is how a stuck transfer turns into an uncancellable request. Output events flow back over a similar channel but in the opposite direction, and their consumer is the user’s connection, so backpressure means pausing or dropping per-request streams rather than blocking the engine. Bulk data — KV regions between prefill and decode workers, encoder embeddings toward the language model — moves over device-to-device paths such as collective transports or direct memory access, because routing gigabytes through a CPU queue would spend Chapter 4’s bandwidth budget on copies. Metadata like block tables rides with the step plan, sized so that serializing it costs far less than the step it describes.

The failure analysis differs by class. Losing a control message loses a decision; losing a bulk transfer loses bytes that may be recoverable from their source; losing an output event loses tokens the caller has already paid latency for. A single unified channel cannot give each class what it needs — which is the whole argument for separating them.

How vLLM and SGLang divide the work

At the pinned source revision, vLLM’s path includes an asynchronous engine, an engine-core client, the engine core, scheduler, executors, workers, and model runners. Useful entry points are AsyncLLM, EngineCore, and the GPU ModelRunner.

SGLang exposes the corresponding work through a TokenizerManager, Scheduler, TpModelWorker, and ModelRunner.

Read the two frontends closely and the same shape appears twice. In vLLM, AsyncLLM.generate() documents its own four-step choreography in its docstring: create an AsyncStream for the request, process the input, add the request to the detokenizer, and hand it to the EngineCore — which runs in a separate process. The method then loops over a RequestOutputCollector, and its read pattern, q.get_nowait() or await q.get(), carries a comment worth internalizing: draining without awaiting avoids task switching under load. The streaming contract ends where the client ends it — if the HTTP caller disconnects, Python raises GeneratorExit, and the method calls abort(request_id, internal=True) so the engine stops doing work nobody will read.

vLLM’s return path runs in one background task, _run_output_handler. It pulls EngineCoreOutputs from the engine core, slices them into chunks bounded by VLLM_V1_OUTPUT_PROC_CHUNK_SIZE — with an explicit await asyncio.sleep(0) between chunks so the event loop can serve other tasks — and passes each slice to output_processor.process_outputs, which pushes finished RequestOutputs onto per-request queues rather than returning them. Two details carry real operational weight. When output processing discovers a stop string, the handler calls engine_core.abort_requests_async for those requests, because the engine core does not yet know they finished. And any exception in the handler reaches output_processor.propagate_error(e) — one background-task failure becomes an error on every live stream, because there is no per-request recovery from a dead output path.

SGLang’s TokenizerManager plays the same role with different machinery. Its rid_to_state: Dict[str, ReqState] is the frontend’s truth about every in-flight request; _send_one_request wraps payload fields for transport and dispatches the tokenized request toward the scheduler. Results come back through a dedicated loop, handle_loop, which receives batches from the detokenizer and routes BatchStrOutput and BatchTokenIDOutput messages into _handle_batch_output. That handler performs a lookup that vLLM handles structurally instead: rid_to_state.get(rid). When the lookup misses, the code does not crash — it logs “Received output for {rid=} but the state was deleted in TokenizerManager,” skips health-check identifiers, and moves on. The race this tolerates is exactly Chapter 3’s overlap hazard: a client disconnected, cleanup ran, and a result computed earlier still arrived. Both systems pay this cost somewhere unavoidable; where they differ is instructive — vLLM keys outputs by request identity end to end, while SGLang keeps an explicit per-request state table at the boundary and defends it against late arrivals.

Do not compare these systems by counting boxes. Compare what crosses each boundary and which component owns the truth. A separate process is meaningful only if you understand the isolation it provides and the communication it adds.

Worked example: classify the waits

The control path for one request is submit, validate, admit, schedule, allocate, execute, and finish. Its data path is text, token IDs, tensors, KV blocks, logits, sampled IDs, and streamed text. Drawing them separately exposes four different waits.

Admission waits before allocation so rejected work cannot consume model state. The runner waits for the scheduler’s block table so attention addresses the right pages. Sampling waits for logits because the next token is a true data dependency. Cleanup waits for the GPU completion event so a live address is not reallocated. These waits protect correctness.

Tokenization for the next request and output processing for the previous step do not necessarily protect those invariants. They can overlap execution if their request-state updates are versioned and queues remain bounded.

The classification generalizes into a review habit. Every wait you find in a server should fall into one of three bins: it protects an invariant (keep it), it protects nothing (remove it), or it protects something cheaper could protect (shrink it). The four waits above give one instance of each kind. Sampling-waits-for-logits is pure invariant: removing it produces tokens from the wrong distribution, so it stays no matter how expensive it gets. Cleanup-waits-for-completion is an invariant that can shrink: instead of one synchronization per request, engines batch releases behind generation counters and reclaim many sequences’ state in one pass. Tokenization and output processing, unversioned, were the accidental bin — work that blocked execution while protecting nothing, fixed by making them overlap safely. Most latency incidents reduce to a correctness wait that grew large enough to notice, or an accidental wait nobody classified at all.

Practice: draw two paths and defend every wait

Trace one request through the frontend, engine process, scheduler, worker, model runner, and output process. Draw messages and state transitions on the control path; draw tokens, tensors, block tables, and logits on the data path.

Mark every CPU/GPU and process/process wait. For each, state the invariant or classify it as an overlap candidate. The worked classification is in Appendix G.

6. Scheduling the Decode Loop

At the beginning of a model step, an engine may have three kinds of work waiting. Several conversations need one more decode token. A new request needs to process a 12,000-token prompt. Another request has a deadline and higher priority.

The GPU cannot execute a vague collection of requests. It needs a concrete batch with valid memory for every position. Building that batch is the scheduler’s job.

The job is harder than picking winners. Every step is a small negotiation among four parties with different clocks: users who notice gaps in their stream, a GPU that wants full and regular batches, a memory pool that fills as conversations grow, and downstream stages whose capacity the scheduler does not control. A policy that pleases one party at step granularity can starve another over a minute. This chapter builds the scheduler one decision at a time — batch membership, work budgets, chunking, queue policy, preemption, and admission — because each decision exists to protect a different one of those parties.

Why static batches waste work

In a static batch, the server groups requests and runs them together until all finish. This works well when inputs and outputs have similar shapes. Text generation is less cooperative.

Continuous batching changes membership at every engine step.

flowchart LR
    Q["Waiting requests"] --> S["Scheduler"]
    A["Active decoders"] --> S
    S --> B["Mixed token batch"]
    B --> E["Execute one step"]
    E --> F{"Finished?"}
    F -->|No| A
    F -->|Yes| O["Output and free state"]

One sequence may stop after five tokens while another continues for 500. The finished sequence leaves an empty slot, but the batch remains alive for the long request. Padding preserves a regular tensor shape while spending compute and memory on positions that no user needs.

The waste has a size. Take a static batch of sixteen sequences whose completion lengths spread evenly between 5 and 500 tokens — a declared assumption, but a realistic one for mixed traffic. At the moment the batch finally finishes, the average member stopped long ago, and average occupancy over the batch’s life is roughly half: about half the token work the GPU performed was padding that no request needed. Chapter 1 walked this cost for a single padded step; static batching multiplies it by keeping the padding alive for the batch’s entire remaining duration, and no scheduler setting can recover it because membership is frozen until the whole batch drains.

Iteration-level scheduling rebuilds membership between model steps. Finished sequences leave and waiting sequences enter. This technique is commonly called continuous batching. The Orca paper showed how iteration-level scheduling improves transformer serving.

Continuous batching makes the GPU busier, but it also creates a fast control loop. The engine must reconsider work and memory every step. That cadence is the chapter’s real subject: a control loop running at engine-step frequency has to be cheap enough to run thousands of times per second, yet complete enough to notice a full memory pool, a missed deadline, or a stalled expert before committing the next batch.

A request count is not a work budget

Suppose the scheduler can admit 32 requests. That limit says little about the next step. Thirty-two decoders need roughly one new token each. One large prefill may need thousands of token positions.

A token budget forces an explicit priority between work types.

flowchart TB
    R["Step token budget"] --> D["Reserve active decode tokens"]
    D --> P["Add bounded prefill chunks"]
    P --> X{"Memory and deadline still safe?"}
    X -->|Yes| E["Execute batch"]
    X -->|No| C["Delay, preempt, or reject"]

Schedulers therefore use a token budget in addition to a sequence limit:

scheduled tokens <= token budget
active sequences <= sequence budget
allocated state <= available capacity

Multimodal and speculative execution add more budgets. The engine may limit encoder work, media items, draft tokens, or the number of requests using a particular adapter.

Tokens are still an approximation. A prefill token with long-context attention can cost more than a decode token. An MoE token may take a different path from its neighbor. The budget is useful because it is cheap to compute, not because all tokens are equal. That cheapness is deliberate: the budget is evaluated every step, so it must cost far less to compute than the step it shapes. Any richer cost model — per-token attention cost, expert-routing estimates — belongs to admission or placement decisions made once per request, not to the per-step loop.

The long-prompt problem

Return to the 12,000-token prompt. Processing it as one prefill operation may occupy a long engine step. Every active conversation pauses while the GPU works on the new prompt. Their average TPOT might remain acceptable, yet users notice a large gap in streaming output.

Chunking plus decode reservation, over three steps of the worked example.

flowchart TB
    subgraph S1["Step 1: budget 16"]
        direction LR
        P1["A prefill chunk: 8"] --- D1["B prefill: 4"]
    end
    subgraph S2["Step 2: budget 16"]
        direction LR
        P2["A prefill chunk: 8"] --- D2["B decode: 1"]
    end
    subgraph S3["Step 3: budget 16"]
        direction LR
        P3["A prefill chunk: 8"] --- D3["B and C decode: 2"]
    end
    S1 --> S2 --> S3

The first diagram is the control loop that replaced static batching: the scheduler re-forms the batch every step, and finished work exits through the decision node instead of holding slots. The second is the budget policy that makes the loop safe under load — decode first, prefill with what remains, then a safety gate. The third shows both working together on this chapter’s worked example: request A’s 24-token prefill flows through bounded chunks while B’s four-token prompt and eight-token output keep a reserved slot in every step, and high-priority C joins as a decoder the moment it arrives.

Chunked prefill divides the prompt across several steps. The scheduler can mix a chunk with decode work, allowing existing conversations to keep moving. The Sarathi-Serve paper develops this idea as a way to control interference between prefill and decode.

Smaller chunks protect decode latency but perform more scheduling and metadata work. They may also lose some efficiency from large matrix operations. Very large chunks recover that efficiency and recreate the stall. The right size depends on the model, batch composition, parallel plan, and SLO.

This is a recurring pattern in inference systems: a parameter that looks like a hardware tuning knob is also a policy about which user waits.

Deriving a chunk ceiling from the ITL budget

The chunk-size trade-off can be made numeric with the planning constants from Chapter 2 and Appendix G: a mixed step’s prefill cost is approximately 20 + 0.035 × chunk tokens milliseconds, and the service owes decoders an inter-token latency of at most 150 ms. A decode-only portion of a mixed step might add around 10 ms for a modest batch — a declared planning assumption. The ceiling falls out of the budget:

20 + 0.035 * c + 10 <= 150   =>   c <= 3,428 tokens

So roughly a 3,400-token chunk is the largest this service can mix into a step without breaking its decode promise. The arithmetic cuts both ways: a team that raises the chunk to 8,192 for prefill throughput has silently rewritten the SLO — mixed steps now take about 317 ms, twice the inter-token budget — and a team that tightens the SLO to 100 ms must shrink chunks to about 2,000 tokens and accept more scheduling overhead. Chunk size is where the throughput-versus-latency exchange becomes a single integer, and the integer should be derived, not borrowed from another deployment’s config.

The same formula prices the chapter’s opening problem. An unchunked 12,000-token prefill takes about 20 + 0.035 × 12,000 = 440 ms — every active decoder sees a single 440-millisecond gap, well past the 150-ms promise. Chunked at 3,400, the same prompt becomes three large chunks and a tail of 1,800 tokens, each mixed into a step of at most about 150 ms: total prefill time barely changes, so the big prompt’s own TTFT grows only by scheduling overhead, while every other conversation’s worst gap drops from 440 ms to the SLO boundary. That asymmetry — one user pays almost nothing, everyone else stops paying a penalty — is the entire argument for chunked prefill in four numbers.

Which request goes first?

First-come-first-served is easy to explain and usually fair by arrival time. It can still let a large request block smaller ones when the work cannot be chunked. Shortest-job policies improve average completion time but require an estimate and can starve large requests. Priority queues protect important traffic but need quotas or aging so low-priority work eventually runs.

Before choosing a queue policy, decide what fairness means. Equal request starts, equal scheduled tokens, equal accelerator time, and tenant-weighted shares produce different schedules. A token-based policy can remain unfair when tokens have different costs because of context length, modality, or expert routing.

Deadlines add another dimension. Work that cannot possibly finish before its deadline may be better rejected than scheduled ahead of requests that could succeed. This is one reason scheduling cannot replace admission control.

Aging deserves one concrete pass, because it is the standard escape from priority starvation. Give each waiting request an effective priority of priority − age × rate: a background request that arrived sixty seconds ago at a decay of one level per twenty seconds now competes at priority three, ahead of fresh priority-five traffic. The rate is the policy: too slow and starvation persists with better optics, too fast and the priority classes merge into FCFS with extra bookkeeping. Whatever the rate, aging must apply to a measurable quantity — arrival time is the honest one; queue position drifts as requests ahead are admitted, and an aged request can watch its effective age reset every time the queue reshuffles.

What happens when memory runs out?

A running sequence consumes more state as it grows. Eventually the scheduler may be unable to allocate the next block.

The cheapest response is to evict cached state that belongs to no active request. If that is insufficient, the engine can wait, preempt a running request, move state to another memory tier, or reject work.

Preemption frees capacity, but the evicted request loses time. If its state is discarded, the engine must recompute the prefix later. If state is swapped, it must move bytes out and back. Frequent preemption is often a sign that admission allowed too many long-lived sequences or that the cache reservation left too little headroom.

Victim choice changes the cost. A recently admitted request may have little computed state to lose. A large request may free many blocks. A low-priority request may be the correct product decision. The scheduler needs an explicit policy rather than an accidental list order.

What a preempted request costs

The recompute-versus-swap choice has a price comparison, and the Atlas constants make it concrete. Take a request preempted with 4,000 tokens of computed context. Recomputing the prefix later costs 0.06 ms per token — about 240 ms of GPU work, all of it competing with paying traffic when the request resumes. Swapping instead moves 4,000 × 320 KiB ≈ 1.22 GiB out and back across a host path moving tens of billions of bytes per second — on the order of fifty milliseconds of transfer, but the bytes occupy host memory for the request’s entire suspension and the round trip consumes PCIe bandwidth that decode steps share.

Neither number dominates universally. Recompute costs GPU time exactly when the engine is busy enough to have preempted; swap costs capacity and bandwidth continuously while the request waits. Short suspensions favor swap, long queues favor recompute — and a queue deep enough to hold many gigabytes of swapped state is itself a signal that admission, not preemption policy, is failing. This is also why preemption frequency is a first-class metric: it is the visible symptom of an admission boundary set too permissively, and Chapter 7’s allocator exists partly to postpone the day it fires.

Keeping the CPU ahead of the GPU

As GPU kernels become faster, the CPU work needed to prepare each step becomes visible. Rebuilding tensors, copying metadata, processing old outputs, and waiting for device results can leave gaps between GPU operations.

A persistent batch keeps stable slots for running requests and updates only what changed. This reduces preparation and helps preserve fixed memory addresses for graph replay.

An asynchronous scheduler goes further. While the GPU executes step t, the CPU prepares step t+1.

CPU: schedule t ---- schedule t+1 ---- schedule t+2
GPU:        execute t ----- execute t+1 ----- execute t+2

The overlap removes idle time, but the CPU is now making decisions with an incomplete view. It may not yet know which speculative tokens were accepted or which request just stopped. It must reserve memory conservatively and attach versions to results. If a request is preempted, an old output may need to be discarded. A block cannot be reused while an earlier step can still write it.

Conservative reservation has a quantifiable price. Planning step t+1 before step t’s completions arrive means reserving blocks for sequences that may finish within the next few milliseconds — at any moment, up to one step’s worth of allocation is committed on optimism. For a step admitting a handful of sequences, that is a tolerable float; for a step that would admit fifty, the stranded reservation can exceed the free list. This is why overlapped engines bound how far ahead they schedule — vLLM’s multiple in-flight batches and SGLang’s result queue both exist to keep the optimism window at one or two steps — and why the bound tightens when speculation inflates the number of tokens each in-flight step might consume.

At the pinned revision, vLLM’s scheduler.py handles token budgets, preemption, encoder work, speculative lookahead, cache connectors, and multiple in-flight batches in one scheduling path. SGLang’s scheduler.py and overlap_utils.py show another approach to overlapping scheduling and execution.

Reading the two schedulers side by side starts with a comment vLLM leaves at the top of schedule(): there is no “decoding phase” nor “prefill phase” in the algorithm. Each request carries num_computed_tokens and num_tokens_with_spec — prompt plus output plus any speculative draft — and the scheduler’s whole job is to assign tokens so each request’s computed count catches up to its target count. That one framing absorbs chunked prefill, prefix caching, and speculation as special cases of the same bookkeeping, and it explains the budgets the method sets up: a token_budget from max_num_scheduled_tokens, a separate input_budget, slots held back for speculative drafting, and an encoder compute budget for multimodal work. The loop schedules running requests first, then admits from the waiting queue while budget remains — decode keeps its reserved share, exactly the policy the second diagram draws.

The same loop shows what preemption costs in bookkeeping. When a request cannot fit, the scheduler picks a victim — under the priority policy, the maximum of (priority, arrival_time); otherwise simply the last entry in the running list — and calls _preempt_request, which frees the request’s blocks, resets num_computed_tokens to zero, and puts the request back at the front of the waiting queue. Two details repay attention. First, the caller restores every budget the victim had consumed — token budget, input budget, draft slots, even encoder compute — so the step can admit replacement work in the same pass. Second, preemption under asynchronous scheduling marks in-flight output as stale: num_stale_output_tokens is set from the tokens still in flight, so results computed before the preemption are tracked and drained rather than silently applied — the version discipline from Chapter 5, appearing exactly where the hazard lives. A long_prefill_token_threshold caps how much of one long prompt a single step may take, making chunking a scheduler-internal fact rather than a caller-visible one.

SGLang reaches similar behavior through different seams. Its event_loop_overlap keeps a result_queue of in-flight (batch, result) pairs and processes the previous step’s results with pop_and_process while forming the next batch — and it carries an explicit disable_overlap_for_batch check, because some batches (certain modes, pipeline-parallel boundaries) must not overlap, and the loop needs a synchronous drain point. Batch formation lives in get_next_batch_to_run, whose most delicate resident is chunked_req: the partially processed long prompt is excluded from the running batch so that only finished requests merge back in, and its previous chunk is stashed into the prefix cache only when it actually produced new KV beyond what was already cached — the code checks extend_range.end against prefix_indices rather than stashing unconditionally. New prefill batches come from get_new_batch_prefill, where a prefill delayer consults current pool usage before admitting more prompt work, and grammar-bound requests wait in their own queue until their constraint machines are ready.

The code is complicated because the interactions are real. Prefix hits, chunking, speculative tokens, remote state, and asynchronous outputs all change what “one more step” means.

Admission protects the scheduler

A scheduler orders work that the service has accepted. It cannot rescue a system that accepts more work than it can finish.

Imagine 120 requests arriving each second while the deployment can complete 100 within the SLO. Keeping the GPU full is not success; the queue grows by 20 requests every second. Eventually almost everyone waits too long.

Admission control uses the estimated work, queue, resident state, priority, deadline, and downstream stage capacity to decide whether a request should enter. Rejecting early may produce more goodput than accepting a request that will time out. The rejection must also create backpressure. Automatic retries without delay can turn overload into a larger burst.

The retry arithmetic explains the warning. At 120 arrivals against 100 completions, twenty requests per second are rejected; if every rejected request retries immediately, next second’s arrivals are 140, then 160 — the rejection itself is generating load at exactly the rate the system cannot absorb. A retry with exponential backoff and jitter converts that loop into a damped one, and a retry budget — capping what fraction of traffic may be retries — bounds it entirely. Load balancers and clients own half of this design, which is why admission is a system contract and not a setting inside the engine.

The 120-versus-100 example also explains why admission belongs to the scheduler’s neighborhood rather than a load balancer far away. Only the engine knows its live state bytes, its preemption rate, and its downstream stage queues — the quantities admission decisions consume. A balancer working from request counts alone will keep sending work into a deployment that has already passed its memory knee, and the first symptom to reach users will be preemption storms, not a clean signal the balancer could have acted on.

Worked example: one token budget, four requests

Give the scheduler 16 token slots per step. Request A arrives with a 24-token prefill; B arrives beside it with four prompt tokens and needs eight output tokens. If A consumes an unbroken prefill, B’s interactive response waits even though both fit within a few steps.

With an eight-token chunk limit, the first step can schedule eight tokens of A and four of B. B can enter decode on the next step while A continues in bounded chunks. Reserving decode slots prevents later prefills from breaking B’s output cadence.

The example is incomplete unless step duration depends on its composition. A step with 16 prefill tokens and one with 16 decodes need not take the same time. Use a measured lookup table keyed by decode batch and prefill tokens; otherwise the simulator merely counts tokens.

The chunk ceiling derived above is the same lesson pointed the other direction: the eight-token limit here is small enough that B’s cadence is never in danger, and large enough that A finishes in three steps. A real deployment picks its limit exactly this way — from the latency it owes its decoders and the prefill rate it must sustain — and then defends the choice with a test that fails when someone raises the limit to chase throughput.

Common scheduling mistakes in production

These are the problems Appendix I’s debugging walkthroughs trace back to scheduling configuration most often:

Each fix has a corresponding measurement in Chapter 23’s methodology. The scheduling chapter’s worked example above demonstrates the first three directly.

Practice: implement and explain a schedule

Simulate A (arrival 0, prefill 24, output 4), B (0, 4, 8), high-priority C (1, 8, 4), and D (2, 20, 2) under a 16-token step budget and 40 units of live-state capacity. Compare FCFS with no chunking, eight-token chunks, and priority plus aging.

Report each step’s contents, TTFT, deadline-qualified goodput, preemptions, and memory state. State when D should be rejected. A worked schedule and scoring rule appear in Appendix G.

7. Memory Management and Local Model State

Suppose a chat request has a maximum context length of 64,000 tokens. Reserving a contiguous KV-cache region for all 64,000 positions would make growth easy, but most requests would finish with much of the reservation unused. Reserving only the current length saves memory, but the region must grow without moving state that the GPU still needs.

The solution used by modern engines resembles virtual memory. Requests see a logical sequence of positions. The engine backs those positions with fixed-size physical blocks that do not need to be contiguous.

The resemblance is more than an analogy. Virtual memory solved the same three problems: how to let consumers believe in a tidy private range while physical storage is fragmented, how to share storage safely between consumers, and how to reclaim it without asking anyone’s permission at the wrong moment. Every mechanism in this chapter — the block table, copy-on-write, reference counts, eviction under live readers — has a direct ancestor in operating-system memory management, which is useful because fifty years of OS practice tells you where the bodies are buried.

Logical tokens, physical blocks

Consider a block that holds 16 token positions. A 35-token sequence needs three blocks. The first two are full and the last uses only three positions. When the sequence grows, the allocator can attach another free block anywhere in the cache.

A block table separates logical sequence order from physical placement.

flowchart LR
    L0["Logical block 0"] --> P3["Physical block 3"]
    L1["Logical block 1"] --> P8["Physical block 8"]
    L2["Logical block 2"] --> P1["Physical block 1"]
    P3 --> K["Paged attention kernel"]
    P8 --> K
    P1 --> K

logical positions:  [0 ........ 15][16 ....... 31][32 .. 34]
physical blocks:          7             2            19

A block table records this mapping. The attention kernel uses the table to find keys and values. Because a request no longer needs one large contiguous region, external fragmentation falls and the cache can support a changing set of sequence lengths.

The PagedAttention paper describes this virtual-memory-inspired design and the original vLLM implementation.

Fragmentation, before and after paging

The win is easiest to see by pricing the alternative. Give every request a contiguous reservation sized for the 64,000-token maximum. At 320 KiB of state per token, each reservation commits about 19.5 GiB regardless of what the conversation actually needs — a healthy 8,000-token exchange uses 2.44 GiB of it, so roughly eight times the reservation sits idle even before short requests are counted. On the deployment Chapter 4 walked, each device of a four-way shard owes its quarter of every reservation, about 4.9 GiB against a KV budget near 35 GiB: seven such reservations consume the pool, whether or not their conversations ever grow. The same budget served fifty-six resident conversations when allocation followed actual length — a factor-of-eight difference in concurrency, purely reservation policy.

Paging attacks both fragmentation modes at once. Internal waste shrinks to the tail of one block — at most fifteen unused positions out of sixteen, under five megabytes per sequence and more than three orders of magnitude below the contiguous case. External fragmentation stops being a scheduling concern: free memory fragmented into scattered single blocks is perfectly usable, because the allocator attaches them one at a time. The residual cost is metadata — larger block tables crossing the scheduler-to-worker boundary every step — which is why block size, the next section’s subject, is a genuine trade and not a knob to minimize blindly.

Choosing a block size

Block size looks like a low-level allocator setting, but it influences the whole engine.

Small blocks waste little space at the end of a sequence and allow fine-grained prefix matches. They create larger block tables and more allocation work. Large blocks reduce metadata and may suit an attention kernel better, but waste more tail space and only reuse prefixes at coarser boundaries.

Transfers add another consideration. Moving many tiny blocks can pay protocol overhead repeatedly. A large transfer unit may move unused positions.

Backend constraints sometimes narrow the choice. SGLang’s current attention-backend guide documents page-size requirements for several implementations and explains the trade-off between kernel performance and prefix-match granularity. The correct size is part of an execution plan, not a universal constant.

The interactions run wider than allocation. Chapter 6’s prefill chunking interacts with block size because a chunk that stops mid-block leaves a partially filled block whose publication timing the lifecycle rules govern; Chapter 15’s cross-node transfers price block size directly, since the transfer unit determines how much protocol overhead repeats per hop. When a team changes block size, it is quietly renegotiating with the scheduler, the kernel, and the transfer layer simultaneously — which is why the setting belongs to the execution plan review, not to a config default.

One worked contrast shows how sharply the choice bites. At sixteen-token blocks, a 100-token conversation fills seven blocks and wastes at most fifteen positions — under five megabytes of tail. At 256-token blocks, the same conversation fills one block and abandons 156 positions: about forty-nine megabytes of state held for thirty-two megabytes of use, waste exceeding the payload. For a chat fleet dominated by short requests, the large block is difficult to justify at any batch size; for a deployment of long documents where every sequence fills dozens of blocks, the tail is rounding error and the smaller tables and kernel-friendly pages win. The block size that maximizes reuse granularity is a function of the length distribution the service actually serves — one more quantity Chapter 2’s workload records exist to pin down.

A block has a lifecycle

Allocation is not merely “free” or “used.” A block can be allocated while a step is about to write it, valid and owned by a running request, valid but kept only for reuse, or waiting for an asynchronous transfer to finish.

Reusable blocks move through ownership states before returning to the pool.

flowchart LR
    F["Free"] -->|allocate| W["Private and writable"]
    W -->|GPU complete| S["Sealed"]
    S -->|publish| R["Reusable and referenced"]
    R -->|evict index| D["Draining"]
    D -->|reference count zero| F

The first diagram is the indirection that makes everything else possible: logical order is a fiction the kernel resolves through the table, so growth, sharing, and release never require moving bytes. The second diagram is the discipline that keeps the fiction honest — a block becomes visible to others only after its writer is provably finished, and leaves the pool only after its last reader is provably gone. The table below names the four moments where that discipline is most often violated, and what each violation looks like from outside.

free -> reserved -> being written -> valid and owned
     -> valid and reusable -> evictable -> free

Asynchronous engines must be conservative. If step t can still write a block, the allocator cannot hand that address to step t+1 for another request. A remote sender cannot release a block while a transfer still reads it. Engines may defer release until the relevant stream or acknowledgement proves the old use has finished.

Reference counts protect ownership. Cache policy decides how long an unowned, valid block remains available for reuse. Combining those two ideas risks evicting state that a live request still needs.

The distinction matters most at transitions. A block moving from private to reusable crosses from “one owner’s truth” to “many readers’ assumption,” and the engine must prove the writer finished — GPU event, stream sync, or acknowledged transfer — before flipping the bit in between. Skipping the proof works in benchmarks, where steps complete predictably, and fails in production exactly when a cancellation, preemption, or stalled transfer makes completion order surprising. This is why the lifecycle diagram’s middle states exist at all: they are the proof obligations, made explicit.

Cancellation exercises every state at once, which is why it makes the best test case. The cancelled request’s in-flight blocks must stay pinned — not published, not freed — until the completion event proves the GPU finished writing them; only then may they seal into reusable state or drop back to the pool. Engines that release eagerly show a characteristic signature: rare garbage tokens in unrelated requests, appearing only under cancellation load, because a reallocated address was still being written by a ghost. The “deferred-release age” signal in the table above is the operational probe for this — how long blocks wait between their request ending and their last use provably finishing — and an age that grows with load is an engine telling you its proofs are falling behind its execution.

Two eviction refinements round out the picture. First, eviction policy operates at block granularity here, but token-level policies also exist: score each cached token by its estimated future importance (attention-magnitude schemes are the canonical example) and drop low-scoring positions while keeping the block. They trade correctness structure for capacity — a dropped token changes attention outputs for everything after it, so unlike block eviction under prefix identity, the result is no longer equivalent to a cache miss. Fine for lossy compression deployments that accept it; wrong for anything that promised Chapter 22-style output equivalence. Second, eviction under adapters must consider the adapter dimension too: a block reused under a different adapter is not a hit at all, which is the subject of the next section.

Serving many adapters at once

Adapter serving is where cache identity becomes a scheduling problem. Take a low-rank adapter at rank 16 on Atlas’s hidden size of 8,192: each layer carries two matrices of 8192 × 16 BF16 values, 2 × 8192 × 16 × 2 = 512 KiB per layer, about 40 MiB across 80 layers — four orders of magnitude smaller than the 140 GB base model. That ratio is the whole economics of adapter-dense serving: a fleet can hold thousands of adapters resident for the cost of one extra base replica, and Chapter 17’s 800 ms cold-adapter load is not I/O wait but the price of not having the weights paged where the batch needs them.

The serving designs follow from the arithmetic. Because an adapter’s working set is tiny and its compute is two thin matrix products per layer, engines keep every adapter resident and batch across different adapters in one step: the base weights are read once regardless, each sequence adds its own low-rank products, and the per-request extra arithmetic is a few percent of the step. The hard parts are the ones this chapter already built for KV state: the activation buffers the low-rank paths need must be paged and sized per batch composition, the block table must carry which adapter each sequence runs under so a mixed step never mixes identities, and CUDA-graph capture (Chapter 9) must either fix the adapter set per graph or read pointers dynamically — a captured graph with baked adapter weights silently serves the wrong model, the same failure class as Chapter 20’s stale-weight caches. When an interviewer asks how one replica can serve a thousand customer-specific models, the answer is this section: adapters make weights a per-request cache problem, and everything from Chapter 7 applies with 40 MiB objects instead of gigabyte ones.

Reusing a prefix

The support assistant in Chapter 1 begins every conversation with the same system prompt. Once the model has processed that prompt, later requests can reuse its KV state instead of repeating the prefill.

But matching text is not enough. The cached state depends on the model and weight version, tokenizer, exact token IDs, positions, adapter, attention configuration, and any multimodal features. A service may also include a tenant namespace or cache salt to prevent sharing across isolation boundaries.

Engines commonly index prefixes in one of two ways. A chain of hashes identifies successively longer blocks. A radix tree stores shared token paths and makes branches explicit. Hash indexing works naturally with page-granular and distributed lookup. Radix indexing makes structured sharing easy to see. Both need collision handling and version separation.

The SGLang paper introduced RadixAttention for reuse across structured language-model programs. In the pinned code, SGLang’s radix_cache.py contains prefix matching, insertion, request caching, and eviction. vLLM’s kv_cache_manager.py coordinates request allocation and cached-block lookup.

Read the two implementations and the identity rules from earlier in this section stop being abstract. In SGLang’s RadixCache, the lookup key is a RadixKey carrying the token IDs plus an optional extra_key, and the match_prefix docstring states the namespace policy outright: entries with identical leading tokens but different extra_key values are “kept disjoint and never share prefix nodes,” which is how LoRA adapters, sampling salts, and cache versions partition the tree without changing the token content. Matching is page-aligned — keys are truncated to a multiple of page_size before lookup, so reuse boundaries obey the allocator’s granularity — and when a match ends inside a stored segment, the method splits that node once to expose a precise boundary, a structural refinement that duplicates no data. The same walk refreshes access timestamps, which is how lookup feeds the configured eviction strategy: using a prefix is itself a cache-policy event. One more detail shows how deep identity goes: the key may be converted to a bigram view when speculative decoding is active, because draft-token patterns change what a reusable segment means — even the matching representation bends to the execution mode.

vLLM’s get_computed_blocks enforces the complementary rule on the consumer side: returned cached blocks must be full, and max_cache_hit_length is set to request.num_tokens - 1. The comment explains why — if every prompt token hit the cache, there would be nothing left to run through the model, and no logits would exist — so the last token is always recomputed. Because allocation is block-aligned, that single recomputed token can drag a whole block with it; the comment flags this honestly as a known inefficiency. The same function shows multi-cache realities surfacing in the interface: the coordinator finds the longest hit per state group, and when sparse-retention groups such as Mamba or sliding-window layers lag behind the full-attention groups, the result carries a shared_prefix_boundary marking the junction all groups can agree on.

Allocation closes the loop in allocate_slots, whose docstring draws the block layout of a request as <comp> | <new_comp> | <ext_comp> | <new> | <lookahead> — already-computed, newly-hit, externally-delivered, to-be-run, and speculative-reserved regions laid end to end. Two of its parameters are admission policy hiding inside an allocator: full_sequence_must_fit forces the whole sequence to fit now, closing the loophole where chunked prefill checks only whether the first chunk fits and strands the rest; and reserved_blocks keeps free blocks aside for in-flight sequences, so an asynchronous KV load cannot consume pages a prefilling request is relying on. The boundary between “memory manager” and “scheduler” runs straight through this signature — which is why Chapter 6 ended by promising this chapter.

Sharing and copy-on-write

Parallel samples or beam candidates can share the blocks that represent their common prompt. When they generate different tokens, their new state diverges. The request shape that triggers this is ordinary: one API call asking for four completions becomes, at Chapter 5’s boundary, four execution requests whose prefixes are identical by construction — sharing is not an optimization the caller requests but a consequence of what the engine notices.

Full, immutable blocks can remain shared. If two sequences share a partially filled block and one needs to write into it, that sequence receives a private copy. This is copy-on-write. It saves memory, but cancellation and completion must update references exactly once.

The same branching appears in multi-turn conversations. A shared system prompt may be extremely valuable; thousands of rare branches may not be. The cache needs an eviction policy, not merely the ability to retain everything.

What branching actually costs

The savings are easiest to trust with numbers. Take four parallel samples of one prompt — 1,000 tokens of shared context. Without sharing, four private sequences hold 4 × 1,000 × 320 KiB ≈ 1.22 GiB; with full blocks shared, the pool holds one copy of 313 MiB, and the three siblings are pure saved capacity. Divergence starts costing only when children write: the first token each child emits lands in the shared partial tail, which must be copied — sixteen positions, about five megabytes per child, once. From there each child pays only for what makes it different: after fifty divergent tokens, a child owns roughly four extra blocks plus the copied tail, tens of megabytes against the original hundreds. The economics that make beam search and best-of-n sampling affordable are exactly these: share everything immutable, pay only at divergence, and let reference counts settle who still needs each block.

Copy-on-write also interacts with the identity rules in a way worth noticing: a branch that copies its tail inherits the parent’s provenance up to the fork point and owns everything after. If the branch later changes adapter mid-life — rare, but tools do this — the inherited portion stays valid only under the original namespace, so the engine must either forbid the change or re-key the branch’s identity from the fork point onward. Systems that skip this check produce caches that serve correct-looking state assembled from two incompatible worlds.

A hit rate can be misleading

Least-recently-used eviction is a reasonable starting point. It does not know how expensive a prefix is to recompute, how likely it is to return, how many bytes it occupies, or whether another tier already holds a copy. Even its central quantity needs interpretation in a tree: when a shared system prompt sits at the root of thousands of branches, every hit below it touches the root, and naive recency bookkeeping makes the root permanently immortal while the leaves — where workload change actually shows first — evict first. The clock is part of the policy, not an implementation detail beneath it.

Imagine two cached prefixes. One contains 10,000 tokens and is reused once an hour. The other contains 100 tokens and is reused every second. A raw request hit rate favors the small prefix. Saved prefill computation may favor the large one. Saved work per byte may produce a third answer.

Useful cache metrics include matched tokens, compute time avoided, bytes held, bytes transferred, eviction churn, and the effect on request latency. Hit rate alone is not enough.

Pricing two prefixes

Give the pair numbers. The 10,000-token system prompt saves about 0.035 ms × 10,000 = 350 ms of prefill work each time it hits; at once an hour, that is 350 milliseconds of GPU time avoided per hour, bought with 10,000 × 320 KiB ≈ 3.05 GiB of residency. The 100-token preamble saves 3.5 ms per hit, but at one hit per second it avoids about 12.6 seconds of prefill per hour while occupying barely 31 MiB. Measured per byte, the small prefix is thousands of times more productive — yet a deployment with spare memory should absolutely keep the big one, because 3.05 GiB sitting idle in an uncongested pool costs nothing and buys 350 ms off every conversation start.

The ranking flips exactly when memory becomes scarce. Under pressure, those gigabytes have an opportunity cost — Chapter 4’s admission walk priced a rank-share of long sequences at roughly 0.61 GiB apiece — and evicting the hourly giant to admit another resident conversation may raise goodput even though the giant’s individual hits feel valuable. That is the real lesson of the misleading hit rate: cache value is a function of current scarcity, not a property of the entry, and any policy fixed at insertion time will eventually be answering yesterday’s question.

A cost-aware policy falls out of the same framing without much ceremony. Score each cached prefix by its expected savings rate — reuse probability × tokens matched × per-token prefill cost, divided by bytes held — refresh the probability from observed hits, and evict lowest score first when the pool needs blocks. The formula’s inputs are all things Chapter 2 said to record: matched tokens, hit frequency, bytes. Two refinements matter in practice: scores must decay, because a workload shift makes yesterday’s hot prefix today’s dead weight; and admission needs the same test as eviction, or the pool fills with newly inserted prefixes that a scoring pass would immediately evict — churn that costs metadata work on every step for zero reuse.

Modern models have more than one cache shape

Some models mix full attention with sliding-window attention. Others include recurrent or state-space layers. They may share KV state between layers or use compressed latent attention.

Each layer type can require a different amount of state and a different retention rule. The longest prefix available for one group of layers may not be valid for another. A correct engine needs a cache specification for each state group and must choose a prefix that all required groups can support.

At the pinned vLLM revision, these ideas appear in kv_cache_interface.py and the cache coordinator. SGLang has separate memory-pool and radix-cache paths for sliding-window, Mamba, and unified layouts. The names will change; the underlying requirement comes from the model.

The coordination cost is subtle: a mixed-model hit is the minimum across groups, so one lagging layer type silently truncates reuse for everyone. A model whose sliding-window group retains four thousand positions caps every hit there, even when the full-attention groups hold thirty-two thousand reusable positions — the window group’s oldest state no longer exists, and no amount of caching in the other groups restores it. An engine that reports prefix hits only from the largest group will flatter itself while requests quietly recompute sliding-window state the report claimed was cached. Honest multi-group metrics count the junction — the boundary all groups reached — not the best single group.

Worked example: publication before reuse

A request produces a final partial block and is cancelled while the GPU write is still in flight. Making that block immediately visible creates two hazards: a reader can observe incomplete data, and cleanup can reallocate an address the GPU still uses.

Keep the block private and pinned until the completion event. Then either seal and publish it under the cache policy or discard it. A branch shares sealed full blocks but copies a partial tail before writing. Eviction removes lookup visibility first and frees storage only after references reach zero.

The two hazards name the two proof obligations precisely. A reader observing incomplete data is a torn-state failure — the block became visible before its writer finished — and the completion event is the proof that prevents it. An address being reallocated under a live writer is a use-after-free failure — the block returned to the pool before its last device use ended — and the pinned-until-event rule prevents that one. Every lifecycle state in this chapter exists to discharge one of these two obligations, and any shortcut that skips a proof is betting that the race it opens never wins.

Content identity also needs a boundary. If token 511 changes, at most the first 511 tokens match. A different adapter or model version invalidates the produced state even when token IDs are identical. Tenant policy may forbid otherwise valid cross-tenant reuse.

Quick KV budget worksheet

Use this worksheet to calculate your deployment’s KV memory budget. Replace the Atlas numbers with your model’s actual constants.

Step 1: Weight memory per rank
  weights = 140 GB (Atlas BF16) / TP_degree
  Example: 140 / 4 = 35 GB per rank

Step 2: Non-KV overhead per rank
  activations ≈ 0.5–1.5 GB (depends on batch size and model)
  graph pool  ≈ 0.5–2.0 GB (depends on captured buckets)
  framework   ≈ 0.5–1.0 GB
  Subtotal:   ≈ 1.5–4.5 GB

Step 3: Available KV memory per rank
  available = GPU_memory - weights - overhead
  Example: 80 - 35 - 3.0 = 42 GB per rank

Step 4: Maximum tokens in KV cache
  KV per token per rank = KV_bytes_per_token / TP_degree
  Example: 320 KiB / 4 = 80 KiB per rank per token
  Max tokens = available / (KV per token per rank)
  Example: 42 GB / 80 KiB ≈ 550,000 tokens

Step 5: Maximum concurrent sequences
  max_sequences = max_tokens / average_context_length
  Example at 4K context: 550,000 / 4,000 = 137 sequences
  Example at 32K context: 550,000 / 32,000 = 17 sequences

Step 6: Set max-num-seqs to 85% of Step 5
  Example at 4K: 137 × 0.85 ≈ 116
  Example at 32K: 17 × 0.85 ≈ 14

The 85% margin prevents preemption storms. If your actual traffic has variable context lengths, use your p90 context length in Step 5. The decision checklist in Appendix D walks this calculation with additional considerations for adapters, speculative decoding, and quantized KV.

Practice: construct a cache safety matrix

Starting from one 512-token prefix, vary exactly one of token content, adapter, image feature, position scheme, model version, tenant, and physical block layout. State the legal reusable prefix and why.

Then test cancellation during a write, branching from a partial block, and eviction with a live reader. Assert unpublished-state isolation, copy-on-write, eventual reference release, and output equivalence with caching disabled. The worked matrix is in Appendix G.

8. Kernels and Attention Backends

The scheduler has chosen 23 requests for the next step. Their sequences have different lengths, their KV blocks are scattered through memory, and some use an attention pattern that others do not. The model runner must turn this irregular description into fast GPU work.

That work is performed by kernels: programs that execute across many GPU threads. A model server may launch hundreds of kernels in one step, including matrix multiplications, normalization, positional encoding, attention, activation functions, expert routing, sampling, and memory copies.

Each kernel is a small contract: it promises a numerical result for a family of shapes, and the engine promises to feed it shapes it can handle. Most of this chapter is about what happens when those promises meet — a kernel that is fast for one shape family and slow for another, a backend that is correct only for certain attention semantics, a fusion that wins in isolation and loses in a step. The runner’s craft is knowing which contract governs the current step, and the benchmark discipline at the end of the chapter exists because no single measurement can check them all.

Why fewer operations can mean more speed

Framework code often expresses a calculation as several tensor operations. Each operation may write an intermediate tensor to high-bandwidth memory, only for the next operation to read it back.

Kernel fusion keeps intermediate values in registers or on-chip memory and performs several logical operations in one launch. A fused normalization and residual update, for example, can avoid round trips through device memory.

Fusion helps when memory traffic or launch overhead is the bottleneck. It can hurt when the combined kernel uses too many registers, lowers occupancy, or prevents a specialized library routine from running. “Fused” is not a synonym for “faster.” It is a claim about a different movement and launch pattern.

The arithmetic-intensity frame from Chapter 4 tells you which case you are in before benchmarking. A fusion that eliminates one intermediate round trip removes bytes from a low-intensity operation — exactly the fix the roofline prescribes below the crossover. The same fusion applied to a compute-bound operation removes bytes the memory system was not waiting on anyway, while the merged kernel’s register pressure may slow the arithmetic that does bind. “Should this fuse?” is “which side of the crossover is this operation on?” wearing an implementation hat.

Pricing one fusion

The launch half of the argument deserves its own arithmetic. Take the fused normalization-plus-residual example and assume the decoder’s hidden state is 8,192 values at BF16 — sixteen kilobytes per sequence-position tensor. Unfused, the pair runs as two kernels: the normalization writes its output, and the residual add reads it back, costing one extra thirty-two-kilobyte round trip and one extra launch per layer. The round trip is trivial against a step’s total traffic. The launch is not: at a few microseconds each, the extra launch costs perhaps 80 layers × 4 µs = 320 µs per step — six percent of Chapter 1’s five-millisecond step, spent doing nothing but starting work. This asymmetry explains why fusion decisions in serving are usually won on launch counts rather than bytes, and why the win grows as steps shrink: the same fusion that saves three percent of a prefill-heavy step saves more of a thin decode step, where fixed overheads are a larger share.

Attention is an I/O problem

The straightforward attention calculation creates a matrix of scores between query and key positions, applies a softmax, and multiplies by values. Materializing the full score matrix moves a great deal of data through GPU memory.

FlashAttention reorganizes the calculation into tiles so that intermediate score regions remain in faster on-chip memory. It computes exact attention while reducing reads and writes to high-bandwidth memory. The important idea is broader than one kernel: algorithm design should count data movement, not only arithmetic operations.

Exactness under tiling is the genuinely clever part. Softmax needs a global maximum over all scores, but a tiled kernel sees one tile at a time — so it carries a running maximum and rescales everything accumulated so far whenever a larger score appears, keeping the result identical to the untiled computation without ever holding more than one tile’s scores. The rescaling is why “tiled attention” and “approximate attention” are different claims, and why the correctness tests below compare against full precision rather than accepting drift.

Serving attention is more complicated than the dense training case. Sequences are ragged. Decode reads a growing history for one new query position. KV state may be paged. Models use different masks, head layouts, latent representations, sliding windows, or sparse patterns. A backend must understand both the model’s attention semantics and the engine’s memory layout.

Counting attention’s traffic

The I/O claim can be priced for one head of the Chapter 3 decoder at a 4,096-token prefill, head dimension 128, BF16 throughout — declared assumptions on tile behavior included. The naive calculation materializes the score matrix: 4,096 × 4,096 × 2 bytes = 32 MiB per head, written once, read back for the softmax, and read again for the value multiply — call it three passes, roughly 96 MiB of high-bandwidth memory traffic per head per layer. The tiled calculation instead streams keys and values through on-chip memory once: about 2 × 4,096 × 128 × 2 bytes = 2 MiB per head, plus negligible query and output traffic. Roughly fifty times less movement for identical arithmetic — and the arithmetic was never the problem, since attention’s intensity sits far above the compute crossover at these shapes.

Decode inverts the lesson. One new query position against a 4,096-token history produces a score vector of eight kilobytes — materialization is trivial — but the kernel must still read the whole history’s keys and values, that same two megabytes per head, to do it. Decode attention is bound by state reads no matter how clever the tiling, which is why Chapter 3’s long-context crossover and Chapter 9’s compression matter more to decode latency than any attention kernel improvement ever will.

Backends are compatibility decisions

An engine may integrate several attention implementations. Selection can depend on device architecture, dtype, head dimension, page size, prefill or decode, mask type, graph compatibility, and parallel plan.

Backend selection is a compatibility decision before a speed decision.

flowchart TB
    R["Runtime shape and model metadata"] --> S["Backend selector"]
    S --> A["Attention kernel"]
    S --> M["Matrix and quantization kernels"]
    S --> E["Expert kernels"]
    S --> P["Sampling kernels"]
    A --> V["Correctness and performance validation"]
    M --> V
    E --> V
    P --> V

If a preferred backend does not support one condition, the engine can reject the configuration or fall back to another path. Silent fallback is dangerous when the operator expects a particular performance profile. Startup logs and metrics should identify the backend actually selected for each layer type.

The danger has a standard failure story. A deployment pins its preferred attention backend; a driver or library update removes it from the supported set; the engine silently falls back to a slower path and keeps serving. No error fires, dashboards stay green because utilization looks normal, and the only symptom is a fifteen-percent throughput decline someone eventually attributes to “traffic changes.” The defenses are cheap: assert the selected backend at startup against an expected value, emit the selection as a labeled metric, and alert when the label drifts. Both pinned engines log their resolution — the discipline is treating that log line as a contract instead of trivia.

Reject-versus-fallback is itself a policy with two failure modes, not a correctness question with one answer. Rejecting at startup turns a missing backend into an availability outage — loud, immediate, safe. Falling back turns it into a slow degradation that may run for weeks. Production services usually want the first for unexpected conditions and the second only for conditions they have benchmarked deliberately, which is why the registry’s override mechanism matters operationally: registering an alternative is how a deployment says “this fallback was chosen,” distinct from whatever the engine guessed.

At the pinned vLLM revision, the attention registry and implementations live under vllm/v1/attention/backends. SGLang centralizes setup in attention_backend_setup.py and maintains device- and model-specific backends elsewhere in the runtime. The number of choices in both trees is evidence that one attention kernel does not fit every serving shape.

The pinned sources show how the selection contract is actually enforced. At the vLLM revision, the backends directory holds roughly twenty implementations side by side — FlashAttention and FlashInfer variants, Triton kernels, Torch’s flex attention, ROCm-specific paths, CPU fallbacks, and a family of linear-attention and Mamba backends for recurrent layer types, plus a dedicated subdirectory for multi-head latent attention. Selection goes through registry.py, where an AttentionBackendEnum maps each name to a default class path — and the design’s most interesting feature is that the mapping is a default, not a constant: deployments can call register_backend() to override any entry at runtime, and a CUSTOM slot exists that refuses to resolve until something registers it. Device gating is explicit in the source — one entry carries a comment restricting it to Hopper-class GPUs — which is the registry telling you that compatibility, not preference, is the first filter.

SGLang’s setup component resolves something subtler than one backend: a pair of them. resolve_attention_backend_strs returns separate prefill and decode backend strings, stamped on the runner before backends are built, so the same model can run one attention implementation while absorbing prompts and another while extending conversations — Chapter 3’s two-kinds-of-work distinction, expressed in the selector itself. The build path then branches on execution mode: disaggregated prefill-mux deployments construct a whole group of decode backends, one per streaming-multiprocessor group, and two-batch overlap wraps the backend in a TboAttnBackend that interleaves two microbatches. A draft worker overrides its own backend string, because target and draft models coexist in one process and cannot share the process-wide choice. None of this is visible in a config file; all of it changes which kernel executes. When a performance profile looks wrong, the first question is which of these resolved paths actually ran — the startup log’s selected backend, per layer type, is the ground truth both engines provide.

Matrix multiplication has shapes, not just FLOPs

Most model compute reduces to matrix multiplication, but two multiplications with equal arithmetic counts can run at different speeds. Dimensions determine whether tensor-core tiles are fully used. Alignment, dtype, transposition, batching, and grouped execution all matter.

The mechanism is visible at the tile boundary. A tensor core consumes tiles of fixed shape — say 128 by 128 by 64 for one common generation — and a multiplication whose dimensions are multiples of those numbers fills every tile; one at 4,096 by 4,096 runs at full efficiency, while 4,000 by 4,000, ninety-eight percent of the work, leaves ragged edge tiles that the hardware pads internally. Two percent sounds tolerable, but decode’s thin shapes are not near-misses — a matrix of thirty-two rows against a 4,096-column weight fills a quarter of one tile dimension, and no autotuner can recover arithmetic the shape never contained. This is why Chapter 6’s batch composition and this chapter’s kernel efficiency are the same conversation held in different rooms.

MoE layers make this visible. Each expert receives a different number of tokens, so the engine often uses grouped GEMM to launch many expert multiplications efficiently. A popular expert has a large matrix; another may receive only a few rows. Padding can improve regularity while doing extra work.

Kernel libraries therefore offer families of implementations. Autotuning measures candidate tiles or algorithms for representative shapes. The result is usually cached because tuning itself is expensive. A production image should decide whether tuning occurs during build, warm-up, or first traffic.

What the tuner knows, and when it learns it

A tuner’s cache is only as good as its key. Entries are keyed by the shape, dtype, and layout family of the call — so a fleet that always serves 4,096-token prefills gets perfectly tuned GEMMs, while one whose contexts drift with traffic pays repeated cold searches on shapes the cache has never seen. On a miss, the library either tunes live — spending the step’s time budget on benchmarking itself, visible as latency outliers at exactly the moments traffic looks new — or falls back to a heuristic choice that may be twenty percent off the tuned optimum. Neither failure appears in a benchmark replay, because replays reuse yesterday’s shapes. This is the operational argument for pinning the workload record from Chapter 2 into the build: tune against the recorded shape distribution, warm the cache at startup, and alert on cache-miss rates in production rather than discovering them as a mystery tail.

Sampling can become expensive

Sampling appears small beside a transformer, but it touches a vocabulary that may contain more than 100,000 entries for every active sequence. Applying penalties, constraints, softmax, top-k or top-p selection, and random sampling through separate kernels creates launches and memory traffic.

Fused sampling paths can help, especially for small models or large batches. Structured-output masks add another tensor operation. The end-to-end effect depends on how much of the step the model itself consumes.

The exposure scales with how little else the step does. Chapter 5 priced the logits copy at about 512 KB per sequence; the kernels that filter and select over those 128,000 entries are individually microseconds, but a step whose model work has shrunk — small batch, short context, quantized weights — can find sampling a visible fraction of its critical path. Chapter 3’s processor chain is the semantic specification; this section’s point is that the chain’s length is also a performance parameter, and fusing it changes launch count without changing distribution semantics — the one optimization in this chapter whose correctness test is a distribution comparison rather than a tensor comparison.

Test the kernel at three levels

A microbenchmark is useful for validating one operation. It controls shapes and removes unrelated work. It does not show whether the engine can present those shapes, whether conversion is needed, or whether the scheduler changes batch composition.

A kernel claim must survive three expanding measurement boundaries.

flowchart LR
    K["Isolated kernel"] --> S["Complete engine step"]
    S --> W["Production-shaped workload"]
    K -. "shape speed" .-> R["Result"]
    S -. "conversion and launch" .-> R
    W -. "queue, cache, and goodput" .-> R

The first diagram is a dispatch, and the selector’s inputs deserve the emphasis: shape and model metadata enter at the top, which means selection is deterministic per configuration — the same model on the same device picks the same backends every start. The second diagram is an epistemology for performance claims: each boundary can invalidate the previous level’s conclusion, and the dashed edges name what each level fails to see. The table below is the same idea as an evidence checklist.

For any proposed kernel change, measure three levels:

1. the isolated operation with representative shapes;
2. a complete engine step containing input preparation and surrounding work;
3. an end-to-end workload with queueing and output processing.

The levels also have different costs and cadences, which decides where each belongs. A kernel microbenchmark runs in minutes and belongs in development — it answers “is this worth pursuing” cheaply and kills most candidates early. The step-level benchmark takes real integration but runs in seconds per configuration, making it the gate for every pull request that touches execution. The workload-level test is the expensive one — replaying production traces with quality evaluation takes hours — so it runs only for candidates that passed the first two, at the moment of enablement. Matching measurement cost to decision size is what makes the discipline sustainable; teams that require level three evidence for every experiment stop measuring altogether.

Before any of it, ask whether the kernel is where the time goes. Chapter 4’s four limits apply here as triage: if the step is bound by host-side gaps or a cross-rank collective, kernel microbenchmarks will show large percentage wins that never reach users — the step’s critical path runs through a different resource. The cheapest level-zero measurement is a step timeline with kernels, gaps, and collectives labeled; if attention does not dominate it, this chapter’s optimizations are the wrong chapter.

Suppose an attention kernel is 20 percent faster in isolation but requires a page size that reduces useful prefix matches. The engine-step benchmark may still improve while a multi-turn workload regresses. All three measurements are necessary to explain the outcome.

Correctness tests should include awkward shapes: one-token decode, long prefill, partially filled pages, uneven head dimensions, empty experts, extreme logits, and masks with no valid continuation. Compare against a trusted implementation with tolerances suited to the dtype. Performance cannot excuse a model-semantic difference.

The awkward shapes earn their place by sitting exactly where fast paths skip work. A fully masked row has a softmax whose denominator is zero — the naive path produces NaN, and the production path must produce a defined token instead; an empty expert receives zero rows, and grouped kernels that assume non-empty groups either crash or corrupt neighbors’ output slots. Each test case is a bet the kernel author made about what never happens, and serving guarantees that something makes it happen eventually — one-token decodes come from max_tokens=1 calls, empty experts from Chapter 3’s skewed routing, no valid continuations from over-tight grammars in Chapter 11’s territory. Tolerances belong to the dtype: a BF16 comparison at float32 strictness fails every correct implementation, and one at float32 laxity hides real semantic drift.

What a launch costs

The three-level discipline earns its keep on overhead that only exists at level two. Chapter 1 walked a step whose five-odd milliseconds of device work carried roughly 0.9 ms of launch overhead when every kernel launched eagerly from Python, and about 0.2 ms under graph capture — launch cost is real, measurable, and shape-dependent. With hundreds of kernels per step, the per-launch microseconds sum into a visible fraction of short steps, which is why fusion, persistent kernels, and whole-graph capture all attack the same tax from different directions. A kernel that wins 10 percent of its own runtime in isolation can lose the step if it forces an extra conversion launch around itself; only the step-level boundary sees the conversion, and only the workload boundary sees whether the step matters.

Worked example: Amdahl meets the page size

Suppose a new attention kernel is 22 percent faster in isolation. Attention is 2.0 ms of a 5.0 ms engine step, so the maximum step saving is 0.44 ms. If the new layout conversion costs 0.3 ms, the actual saving is 0.14 ms, or 2.8 percent—not 22 percent.

Now suppose the kernel requires 64-token pages instead of 16-token pages. More tail waste and coarser prefix boundaries reduce cache capacity. Preemption or recomputation can erase the remaining step win. The three levels answer different questions: whether the operation improved, whether the step improved, and whether users received more qualifying work.

The example generalizes into an enablement rule worth writing down before the benchmark runs: enable if step-level saving exceeds a threshold and workload goodput does not regress and output equivalence holds within tolerance. Writing the rule first prevents the common failure of running the workload test until a favorable window appears. Conditional rules also encode the honest outcome — this kernel helps long contexts and hurts prefix-heavy fleets — which a universal winner claim cannot.

Amdahl’s arithmetic behind the first paragraph is worth keeping in reusable form: a kernel that gets faster by fraction s, running inside a step where it occupies fraction f of the time, improves the step by at most f × s — 0.36 in the worked case, before conversion costs eat their share. The reason serving needs levels beyond this formula is the interaction term Amdahl cannot see: the page-size change altered other components’ behavior (cache capacity, preemption), so the system’s response is not a sum of local speedups. Any optimization that changes shared state — layouts, page sizes, memory reservations — must be judged at the level where its side effects live.

Practice: decide whether to enable the kernel

Evaluate the candidate above at isolated operation, complete step, and production-trace levels. Include batch 1 and 32, contexts 127 and 4,096, partially filled pages, and a multi-turn trace with prefix reuse. Measure conversion, metadata, cache occupancy, preemption, output equivalence, and goodput.

Write a conditional enablement rule rather than declaring a universal winner. See Appendix G for the worked arithmetic.

9. Compilation and Graph Execution

Chapter 1 walked a service that lost. Graph replay cut launch overhead from 0.9 ms to 0.2 ms per step, yet users waited longer, because padding the batch up to the captured size added 1.2 ms of GPU work to a step that had only been 5.2 ms long. Nothing in that story was a bug. Replay did exactly what it advertised. The service regressed because the artifact was chosen without the workload in hand.

That decision is this chapter’s subject. A decode step may finish on the GPU in less time than the CPU needs to prepare and launch all of its kernels. When that happens, a faster kernel does not keep the GPU busy; the empty spaces between kernels become the bottleneck. Compilation and graph execution attack those spaces by doing more planning before the request arrives — but every artifact they produce must be paid for in warm-up time, memory, and padding, and each artifact is only worth its cost for the shapes that actually arrive.

Eager execution pays as it goes

In eager execution, the framework encounters operations and dispatches them at runtime. Each operation walks the same host path: the Python call enters the dispatcher, the dispatcher selects an implementation — the registry work of Chapter 8 — arguments are checked and marshaled, and a launch is issued. None of this work depends on the request content. The same shapes arrive thousands of times per second, and the host answers them identically each time.

Compilation moves repeated host work into reusable artifacts.

flowchart LR
    E["Eager Python and dispatch"] --> O["Operation launches"]
    O --> G["GPU execution"]
    T["Captured or compiled graph"] --> R["Graph replay"]
    R --> G
    S["Runtime shape"] --> D["Artifact dispatcher"]
    D --> T
    D --> E

The arithmetic explains why the gap exists at all. A large transformer runs roughly a dozen kernels per layer; at eighty layers that is near a thousand launches per step. At about a microsecond of host work per launch — a reasonable planning figure, not a measurement — the host owes the GPU roughly a millisecond per step, which is the same order as the 0.9 ms of launch gaps Chapter 1 measured. The GPU work in a decode step does not shrink when the batch is small, but the number of launches is fixed by the model, so the gap hurts most exactly when utilization is already poor.

A compiler captures a region of model computation and transforms it before execution. It may fuse operations, generate specialized kernels, remove redundant work, or choose layouts. PyTorch’s official torch.compile documentation describes full-graph and region-based capture, dynamic shapes, specialization, and debugging options.

Compilation has an up-front cost. If every request shape produces a new specialization, the service can spend more time compiling than it saves. A deployment needs a policy for dynamic dimensions and a cache for reusable artifacts — the same shape-keyed cache discipline Chapter 8’s autotuner already demands.

Where a microsecond of dispatch goes

The per-launch host cost is not one number but a stack of small ones, and knowing the stack tells you what compilation can and cannot remove. Assume a declared breakdown for one eager operation: a few hundred nanoseconds for the Python call frame and attribute lookups, a similar slice for the dispatcher’s pattern matching, a comparable slice for implementation selection and dtype or device checks, and the remainder for argument marshaling plus the driver call itself. Summed, the stages land near the microsecond figure used above — and only the last stage is irreducibly necessary at step time.

Graph execution collapses the stack rather than shrinking each layer. Implementation choice happened at capture; argument layout was fixed at capture; the thousand launches became one replay call. What survives is the host work around the graph — batch assembly, block tables, sampling decisions — which is exactly the work Chapters 5 and 6 built processes and overlap around. This is why graphs and scheduling overlap compose instead of competing: replay removes launches from the critical path while overlap moves the remaining host work off it.

CUDA Graphs capture launches

A CUDA Graph records a sequence of GPU operations and their dependencies, then replays that sequence with much lower CPU launch overhead. NVIDIA’s CUDA Programming Guide separates graph use into definition, instantiation, and repeated execution. Capture runs the region once on a side stream while the driver records the structure; instantiation turns the record into an executable; replay submits the whole structure with a single launch call. A thousand launches become one.

Replay works because much of the operation structure is known. That creates constraints, and each constraint exists for a concrete reason:

• Shapes must fit the captured graph. Kernel launch geometry is part of the record, so a replayed graph cannot grow its batch. Hence buckets.
• Memory addresses must remain stable. The executable bakes in pointers to its intermediate buffers. Paged KV cache helps here — Chapter 7’s block tables give every sequence a stable home — but the activation workspace must persist for the process lifetime, which is why graph memory cannot be returned to the allocator between steps.
• Host-side control flow cannot appear inside a captured region. An early exit or a data-dependent branch executes on the CPU, and the CPU is not recorded. Dynamic decisions must move outside the graph or become device-side work.
• Collectives need special handling. Tensor-parallel all-reduces inside a captured region require communication libraries that support graph capture; a collective that synchronizes ranks from the host breaks the record.

An inference engine usually captures several shapes rather than one. At runtime, it chooses a graph that can cover the active batch and pads or routes unmatched work to eager execution.

Padding versus too many graphs

Imagine capturing graphs for batch sizes 1, 2, 4, 8, 16, 32, and 64. A batch of 23 requests can use the size-32 graph with nine padded slots. Capturing every possible size would avoid padding but consume more warm-up time and graph memory.

Graph buckets trade artifact count against padding and fallback.

flowchart TB
    B["Requested batch size"] --> X{"Compatible captured bucket?"}
    X -->|Exact| R["Replay exact graph"]
    X -->|Larger bucket| P["Pad and replay"]
    X -->|None| E["Compile or eager fallback"]
    R --> M["Record dispatch outcome"]
    P --> M
    E --> M

This is a bucketing problem. Dense buckets reduce wasted work and increase the number of artifacts. Sparse buckets reduce artifacts and increase padding. Traffic distribution determines the right compromise. With power-of-two buckets, the worst case is just under double: a batch of 17 replays the 32 bucket and pads 15 rows, 88 percent extra batch. The average case depends on where traffic actually lands, which is why the practice exercise at the end of this chapter hands you a distribution rather than a single number.

The same issue applies to token counts, prefill chunks, speculative lengths, multimodal shapes, and expert-routing capacity. “Enable CUDA Graphs” is only the start of the execution plan.

Pricing the bucket set

The bucket set’s price has three components, and the first is memory. Capture allocates intermediate buffers through the caching allocator, and replay requires those addresses to stay valid forever, so the pool persists for the process lifetime. Assume the largest decode bucket’s capture grows the pool by 2 GiB — a declared planning figure. If each bucket grew its own pool, five buckets would hold 10 GiB hostage. Engines avoid this by capturing in a deliberate order: vLLM’s get_capture_descs in vllm/v1/cudagraph_dispatcher.py sorts descriptors by (num_tokens, num_active_loras) descending, with the stated intent of memory efficiency. The largest capture grows the pool once; every smaller capture afterward allocates from the space already freed. Expect the total to land near the largest bucket’s footprint, not the sum.

The second component is warm-up time, and it multiplies faster than bucket count suggests. Keys are the cross product of everything the graph depends on: vLLM builds its keys with product(cudagraph_capture_sizes, lora_cases), where the LoRA axis is a single case without adapters, or [0] + captured_counts when adapter counts are specialized. Assume 0.7 seconds per capture and four LoRA cases over five sizes: twenty graphs, roughly fourteen seconds of warm-up before the first request is admitted. Speculative decoding multiplies along a different axis — uniform_decode_query_len = 1 + num_speculative_tokens means each decode slot carries one token per draft plus the base token (Chapter 11), so buckets must align to that stride.

The third component is the smallest: the instantiated executable itself is MiB-scale, not GiB-scale. It still adds up across dozens of graphs, and SGLang attacks exactly that term — the second guided reading below shows how.

Full, piecewise, and breakable graphs

A full graph captures the entire model step. It offers a simple replay path but fails when any region is too dynamic or incompatible.

Mode dispatch tries the strictest key first and relaxes toward eager.

flowchart LR
    K["Batch descriptor"] --> D{"Dispatch"}
    D -->|"exact FULL key"| F["Full-graph replay"]
    D -->|"relaxed PIECEWISE key"| P["Piecewise replay"]
    P --> S1["Captured segment"] --> A["Attention boundary"] --> S2["Captured segment"]
    D -->|"no matching key"| E["Eager execution"]

A piecewise graph divides the model at deliberate boundaries. Static regions use graph replay while dynamic operations run between them. A breakable graph uses a similar idea but treats selected regions as allowed breaks within a larger execution plan.

These approaches matter for modern serving workloads. An MoE router may produce dynamic expert counts. A custom attention backend may not support capture for a particular mode. A multimodal encoder may have dynamic dimensions. Keeping the rest of the model in graphs preserves much of the launch benefit.

At the pinned snapshots, vLLM implements compilation passes and piecewise or breakable graph machinery under vllm/compilation — piecewise_backend.py, breakable_cudagraph.py, and partition_rules.py name the three concerns directly. SGLang contains full, piecewise, and breakable runners under runner_backend as full_cuda_graph_backend.py, breakable_cuda_graph_backend.py, and tc_piecewise_cuda_graph_backend.py, all behind one BaseCudaGraphBackend interface whose methods are capture_one, can_run, and replay. Their coexistence reflects a practical truth: serving graphs need controlled escape routes for dynamic work.

Choosing where a graph may break

The boundary list is the real configuration decision in piecewise execution. vLLM names it splitting_ops: the operations at which the compiler is allowed to cut the model into captured segments. Attention sits on that list because it is the one region whose shape genuinely varies — sequence lengths differ per request, backends own their kernels (Chapter 8), and some modes refuse capture outright. The dispatcher’s constructor enforces the pairing at startup with an assertion: piecewise cudagraph modes require that attention is compiled piecewise or that breakable graphs are enabled, and the assertion message prints the three settings involved (cudagraph_mode, compilation_mode, splitting_ops) so a misconfiguration names itself.

Every additional break returns launch overhead to the step; every removed break risks a capture failure or forces dynamic work into a padded shape. MoE routing is the second common boundary candidate — expert counts are not known until the router runs — so an MoE deployment may carry two break points where a dense model carries one. The practical rule follows from Chapter 8’s compatibility thinking: put a boundary wherever a component’s capture support is uncertain, and nowhere else. Boundaries are load-bearing walls, not decoration.

Guided reading: how vLLM dispatches a step

The dispatcher that decides between those outcomes lives in vllm/v1/cudagraph_dispatcher.py, and its docstring states the contract plainly: the keys it holds “are the only source of truth for valid cudagraphs that can be dispatched at runtime.” The wrappers that would replay a graph do not second-guess the choice; they trust the dispatched mode or pass through to eager.

Three details reward close reading. First, keys are initialized late. initialize_cudagraph_keys carries the comment that it “should be called only after attention backend is initialized,” because only then is the final CUDAGraphMode known — Chapter 8’s backend resolution feeds this dispatcher, and until it resolves, the dispatcher’s mode defaults to CUDAGraphMode.NONE. Startup order is a dependency chain, not a formality.

Second, the bucket map is a precomputed round-up table. _compute_bs_to_padded_graph_size builds a flat list from every batch size up to the maximum; a size that lands exactly on a bucket maps to itself, and everything between maps up to the next bucket. The same method then validates compile_sizes: a compile size that padding would change raises a ValueError telling the operator to use values from cudagraph_capture_sizes. A shape that must not drift is refused at startup rather than silently rounded.

Third, dispatch relaxes in one direction only. It rejects immediately when num_tokens exceeds max_cudagraph_capture_size, returning NONE. It checks the FULL mode with the exact descriptor first — the code comments that “FULL mode needs exact num_reqs because FA3’s scheduler_metadata computation depends on it” — and only then relaxes the descriptor with num_reqs=None, uniform=False to look for a PIECEWISE key, a search the docstring describes as dispatching “a uniform batch to a graph that supports a more general batch.” LoRA adapter counts round up the same way: with specialization enabled, bisect_left over captured_lora_counts finds “the smallest captured num_active_loras that is >= the current.” Strictest first, relaxation as fallback, eager as the floor.

Guided reading: SGLang graph backends and executable dedup

SGLang’s runner_backend package separates the policy from the mechanism. BaseCudaGraphBackend is a deliberately thin interface — capture_one, can_run, replay, cleanup — and the three implementations differ in what they capture, not in how callers invoke them.

The interesting mechanism is in cuda_graph_dedup_mixin.py, gated behind the SGLANG_ENABLE_CUDA_GRAPH_DEDUP environment flag. The observation: many captured graphs are structurally identical — same kernels, same launch geometry, same dependency order — differing only in which buffers they read and write. Yet each instantiates its own executable. The mixin makes structure, not shape, the sharing key. graph_signature walks the raw graph’s nodes and edges through the driver API, topologically sorts them (asserting “CUDA graph contains a dependency cycle” if the sort fails to cover every node), and returns the ordered node payloads plus the sorted edge list. A kernel’s payload is its name, grid dimensions, block dimensions, shared-memory size, and launch attributes — deliberately excluding its arguments. Data pointers are not part of the signature, which is precisely why two graphs over different buffers can match.

Registration then exploits the match. The first graph of a signature instantiates two executables: the live graph_exec and a compat_exec probe. When a second graph with the same signature arrives, register calls cudaGraphExecUpdate against the probe — proving compatibility without disturbing the live executable — and adds the graph to the group. seal tears the probes down at the end of capture and logs the payoff as “captured %d CUDA graphs, deduped to %d execs.”

Replay pays the flip side. If the requested graph is not the group’s current_raw_graph, replay first runs cudaGraphExecUpdate to repoint the shared executable’s parameters at this graph’s buffers, then launches. Assume that update costs on the order of tens of microseconds of host work: invisible against a multi-millisecond step, but real, and incurred every time traffic alternates between members of a group. The trade is one executable’s memory instead of N, bought with a small per-switch update. And the mechanism degrades honestly: if the driver bindings are unavailable or the installed PyTorch lacks raw_cuda_graph, build_deduped_cuda_graph returns None and the engine runs “using plain executables” — an optimization that can fail without blocking startup.

Warm-up is part of deployment

Compilation, autotuning, memory allocation, and graph capture often happen on the first few representative shapes. Sending user traffic during this period creates cold-start latency and can expose untested memory peaks.

A production warm-up should exercise the shapes, precisions, adapters, attention backends, parallel groups, and structured-output paths expected in traffic. It should also respect the deployment’s memory ceiling, and order matters here: capturing a large graph after allocating the entire KV cache can fail even when both would fit under a different reservation order, because capture wants its pool contiguous in time if not in address space. The same reasoning that makes Chapter 7 admit a sequence only when its blocks fit makes a warm-up plan sequence its allocations deliberately.

Artifacts need version keys. Model weights, engine code, compiler version, device architecture, kernels, and configuration can all make an old artifact invalid. A cache keyed on too few of these serves stale code silently — the failure mode is not a crash but a slow, unexplained regression, the same signature Chapter 8’s tuner cache guards against. Key on everything that influenced the artifact, and treat an unkeyed influence as a bug in the key.

Diagnose before disabling

When graph or compiler performance disappoints, separate four cases:

• compilation time is appearing in the measurement;
• shapes are recompiling or missing graph buckets;
• execution falls back to eager mode;
• graph padding or memory constraints outweigh launch savings.

Each case has its own metric: compile count and time, exact-bucket hit rate, fallback rate, and padding ratio respectively. Use compiler logs, graph-dispatch metrics, and a GPU timeline. Compare cold, warm, and steady-state runs — a number that includes warm-up is answering a different question than one that does not. Record how many unique artifacts were created and how often each one served real work.

For an experiment, choose a workload with variable batch sizes. Measure eager execution, compiled eager execution, and graph replay. Report CPU preparation time, GPU gaps, padding, warm-up time, graph memory, and SLO-qualified goodput. A graph mode has succeeded only if it improves the service after its full cost is included — the lesson of Chapter 1’s losing replay, applied as a measurement discipline.

Worked example: bucket 9 is really bucket 16

Suppose captured decode buckets are 1, 4, 8, 16, and 32. A batch of nine replays the 16 bucket and executes seven padded slots. Walk the batch-of-eight case first, where nothing pads. Eager execution spends 1.1 ms of CPU launch work; graph replay spends 0.2 ms; dispatch and padding bookkeeping add 0.15 ms. The net saving is 1.1 − 0.2 − 0.15 = 0.75 ms, so a 5.1 ms step becomes 4.35 ms. Every term is measurable on a timeline, and the sum is the whole argument.

Batch nine is the interesting case. Replaying the 16 bucket still saves the same 0.75 ms of host time, but now seven padded slots execute device work that serves no request. The step loses exactly when that padded work exceeds 0.75 ms — and the padded work can be estimated from a slope rather than guessed. Chapter 1’s walked example measured padding at 1.2 ms for five added rows, about 0.24 ms per row; at that slope, seven rows cost roughly 1.7 ms and the replay loses by nearly a millisecond. But the slope is a property of the workload, not a constant: in a weight-bound decode step the dominant traffic — reading the weights — does not scale with batch size at all, so per-row cost can be far lower than 0.24 ms. Measure the slope on your own step timeline before predicting which side of 0.75 ms batch nine lands on.

Record requested shape, replayed bucket, padding ratio, fallback, and artifact identity on every step. A histogram of requested batch sizes tells you whether to add a bucket or accept eager execution for a rare gap. Compilation time is a startup measurement, not something to hide inside or silently exclude from a steady-state number.

Practice: design the bucket set

Use the batch distribution 1: 8%, 2–4: 17%, 5–8: 31%, 9–16: 29%, 17–32: 15%. Compare eager, compiled eager, and graph replay with buckets 1, 4, 8, 16, and 32. Report cold start, CPU time, GPU gaps, padding, fallbacks, graph memory, and goodput.

Propose one bucket change under a fixed graph-memory budget and explain which traffic it helps. The worked analysis is in Appendix G.

Compilation changes how operations run. Quantization, the subject of Chapter 10, changes the representation of the values they process.

10. Quantization, Precision, and Determinism

A model’s weights may occupy hundreds of gigabytes in a 16-bit format. Storing the same number of values in 8 or 4 bits can make the model fit on fewer devices and reduce the bytes read during decode. That sounds like an automatic win.

The catch is that fewer bits represent fewer distinct values. Quantization must preserve the information the model needs, and the hardware must have an efficient way to use the chosen format. Both conditions are load-bearing. A format the target GPU cannot execute natively can end up slower than the 16-bit original; a format that passes every performance test can still change what the model says. This chapter treats quantization as a systems decision with a quality gate, not a compression setting.

Values, ranges, and scales

Floating-point formats divide their bits among sign, exponent, and significand. Integer quantization usually maps a range of real values onto a small set of integers using a scale and sometimes a zero point. A scale says how wide each integer step is; a zero point says where the real value zero lands inside the integer range. Symmetric schemes skip the zero point and spend everything on step width.

Quantization inserts representation changes into the execution path.

flowchart LR
    W["High-precision weights"] --> Q["Quantize and store scales"]
    Q --> K["Supported low-precision kernel"]
    A["Activations and KV state"] --> C["Calibrate or scale"]
    C --> K
    K --> O["Output logits"]

One scale for an entire tensor is cheap but must cover outliers. Per-channel or per-group scales adapt to smaller regions and preserve more detail, at the cost of extra metadata and conversion work. Dynamic schemes calculate scales from the current activation or token; static schemes use values determined during calibration. Static scales make execution cheaper and reproducible; dynamic scales track the data at the price of a reduction before every quantized operation.

The granularity becomes part of the kernel. Scales are not annotation — they are extra operands the kernel loads alongside every tile, and the group size determines the weight layout in memory. A file labeled “4-bit weights” does not fully describe how groups, scales, outliers, and accumulation are handled.

What calibration actually stores

For static schemes, calibration is the process that decides those scale operands, and it is worth knowing what it produces. A calibration pass runs representative prompts through the unquantized model and records, per tensor, the range the activations actually take. A percentile-based scheme keeps the range covering, say, 99.9 percent of observed values and clips the rest — accepting error on outliers in exchange for a finer step for everything else. The artifact is small: a scale (and optionally a zero point) per quantized tensor, serialized alongside the checkpoint. For a 70B model the calibration file is megabytes against hundreds of gigabytes of weights, but those megabytes change every downstream number.

Two failure modes follow directly. Distribution shift is the first: scales frozen on English chat traffic may clip badly when the service later serves code, tables, or another language, because the outlier structure of activations is language- and domain-dependent. The second is subtler — calibration is measured on activations, but weight-only schemes never see activation statistics at all, so the two families fail differently: weight-only quality degrades smoothly with coarser groups, while activation-quantized schemes can fall off a cliff when an uncalibrated outlier channel appears. SmoothQuant’s insight is aimed exactly here: rescaling activation channels into the weights — dividing activation ranges by a factor and multiplying the corresponding weight columns — evens out the range mismatch so one shared 8-bit scale can serve both sides.

One tile through a 4-bit kernel

Make the representation loss concrete with one group. Take 128 consecutive BF16 weights — group size 128, a common choice — and suppose the largest magnitude among them is 0.5, a declared assumption for the sake of the walk. A symmetric signed 4-bit format offers the integers −8 through 7, so the scale must map +7 to 0.5: one step is 0.5 / 7 ≈ 0.071. A weight of 0.32 becomes round(0.32 / 0.071) = round(4.5), landing on integer 4 and reconstructing 0.286. The stored integer is half the size, and this one weight is wrong by 0.034, about eleven percent of its value. Every weight in the group carries up to half a step of rounding error, and the tighter the group’s range, the smaller the step and the error — which is the entire argument for groups.

A quantized multiply is a real multiply plus a scale, a round, and a wider accumulator.

flowchart LR
    R["Real weights (BF16)"] --> G{"Pick one scale per group"}
    G --> I["Store integers (4/8 bit)"]
    G --> S["Store scales (FP16)"]
    I --> K["Kernel loads both"]
    S --> K
    K --> DQ["Dequantize: w ≈ scale × int"]
    DQ --> ACC["Accumulate in wide type"]
    R -.->|"rounding loses outlier detail"| I

Two properties of that error matter downstream. First, it does not wash out in the dot product: rounding errors across a 128-weight accumulation behave like independent noise, adding in quadrature alongside the signal, so a longer summation does not improve the ratio. The levers are a smaller step (more bits) or a tighter group (better-fitted scales), never more arithmetic. Second, the error is systematic per group — an outlier that stretched the scale taxes every other weight in the group, which is why outlier-aware methods exist.

The metadata has a price too. One FP16 scale per group of 128 adds 2 bytes to the 64 bytes of 4-bit payloads: about 3 percent overhead, lifting 4-bit to roughly 4.12 effective bits per value. At group size 64 the overhead is 6 percent. Finer granularity buys accuracy with bytes and with kernel-side conversion work.

Three families of weight-only methods

Round-to-nearest with good groups — the tile walk above — is only the baseline. The field’s weight-only methods divide into three families by how they fight that systematic per-group error, and the names are worth knowing because the trade-offs travel with them:

Both calibrated families spend a calibration pass to buy back accuracy at fixed bit width, and both inherit the distribution-shift risk described above: their statistics describe the calibration set, not the model. The practical interview-grade summary: at 8 bits the families converge and the format choice dominates; at 4 bits the family choice is worth more than the group-size dial, and the honest evaluation is Chapter 23’s — same trace, same quality gates, differences classified before conclusions.

What can be quantized?

Weight-only quantization compresses model parameters while keeping activations at a wider precision. It directly reduces weight memory and can help memory-bound decode — Chapter 3 showed decode traffic is dominated by weight reads, so halving weight bytes attacks the dominant term. The kernel must unpack or dequantize weights while multiplying them, which is why weight-only formats live or die by kernel support at the shapes that arrive.

Weight-and-activation quantization reduces both operands and can use lower precision matrix hardware. Activations are harder because their ranges change with tokens and layers. Techniques such as SmoothQuant move some activation difficulty into the weights to enable practical 8-bit execution.

KV-cache quantization reduces the persistent bytes per token, allowing more or longer sequences. Attention must dequantize or operate on the compressed state every step. Small quality errors can accumulate over long contexts, so long-sequence evaluation matters.

Communication can also be quantized. Reducing collective or transfer bytes may help a network-bound plan, but conversion and cross-rank numerical behavior become part of the contract.

The final operations are quantizable too — logits and sampler state — but they sit on the thinnest ice. The vocabulary projection is small relative to the model, so the byte savings are minor, while the risk lands on the output contract itself: token probabilities, ranking, and derived quantities such as perplexity or confidence scores all shift with rounding this close to the user. Most deployments quantize everything below the logits and leave the last miles wide.

KV state at half the bytes

The Atlas constants make the KV trade walkable. At BF16, one token of KV state costs 320 KiB and an 8,000-token sequence holds 2.44 GiB. FP8 KV state halves both numbers: 160 KiB per token, about 1.22 GiB per sequence. Chapter 4’s admission budget — 35 GiB of KV space per rank after weights and reserve — then admits roughly twice as many 8,000-token sequences for the same memory: about 114 where BF16 admitted about 57. The per-step cost falls too: the KV read that Chapter 3 counted as a decode step’s second-largest traffic term halves, so attention over long contexts gets proportionally cheaper.

The risks are as concrete as the gains. Attention scores drift as compressed state accumulates over thousands of tokens, and the drift is invisible at short lengths — only a long-context evaluation can see it. Support is a registry question, not a flag: the attention backend selected in Chapter 8 must handle scaled KV state, and the checkpoint must carry the scales under names the loader recognizes. That second hazard is real enough that vLLM’s quantization interface ships a dedicated name-mapping table for KV scales — the guided reading below walks it.

Smaller does not always mean faster

Assume a 4-bit model uses half the weight bytes of an 8-bit model. It may still run slower if the target GPU lacks a native kernel for the format, if group shapes are poorly aligned, or if conversion overhead dominates a small batch. The 4-bit representation might also require a workspace that reduces KV-cache capacity.

Performance depends on a chain:

model format -> engine loader -> quantization method -> kernel
             -> device support -> actual workload shapes

Break any link and the engine may reject the model, fall back to a slower path, or silently convert to another representation. The chain explains a common field surprise: two engines report different throughput for the same 4-bit checkpoint, because “same format” resolved to different methods and different kernels on each. The format name travels with the model; the execution path does not.

Walk one failure to see the two shapes it can take. A checkpoint whose method declares a minimum compute capability of 90 arrives at capability-80 hardware: the registry’s device gate refuses at load time, the service fails loudly before serving, and the fix is a different artifact — the good outcome. The same checkpoint on capable hardware but with an unsupported group size takes the other shape: the method loads, then the kernel layer discovers at first use that its fast path rejects the shape and falls back per call. Nothing errors; the deployment simply runs slower forever. Loud failures get fixed in minutes, silent ones in weeks — which is why the startup-time gate is worth more than its assert.

The workspace tax

Some low-precision kernels need scratch space beyond their operands — workspace for repacked tiles, staging buffers for dequantized copies, or algorithm-specific storage. Assume a declared 3 GiB workspace for a candidate 4-bit kernel. Against Chapter 4’s per-rank budget the tax is immediate: 35 GiB of KV budget becomes 32 GiB, which at 625 MiB per sequence is five fewer 8,000-token sequences admitted. A format that looked free has paid for itself partly in context capacity.

Batch shape taxes it further. Weight-only kernels amortize conversion work across the batch: at batch 32 one dequantized tile feeds many rows, but at batch 1 the same conversion serves a single row, so fixed kernel overheads dominate and the effective bandwidth drops below what the byte count promises. This is why the worked example’s first diagnostic question is “test batch-1 decode kernels” — the small-batch regime is where weight-only formats most often fail to deliver their theoretical traffic savings, and interactive ITL lives exactly there.

Both implementation snapshots contain large quantization registries because formats interact with devices and layer types. vLLM’s implementations live under layers/quantization, while SGLang’s live under srt/layers/quantization. The directories are compatibility maps, not interchangeable labels.

Guided reading: the quantization registry interface

vLLM’s base_config.py splits the problem into two abstract classes. QuantizationConfig faces the checkpoint: get_config_filenames names the files to search for in the model directory, from_config builds the config from the checkpoint’s JSON, and get_min_capability states a hardware floor — its docstring is explicit that the requirement exists because of “the custom CUDA kernels used by the quantization method,” citing capability 70 for Volta, 75 for Turing, 80 for Ampere. The device gate Chapter 8 built for attention backends is baked into this interface at the same depth.

QuantizeMethodBase faces execution: create_weights allocates the layer’s parameters in the format’s own layout, and apply runs the forward pass. Between them sits the dispatch that makes registries necessary: get_quant_method(layer, prefix) returns a different method per layer — or None for layers the format does not quantize. Embeddings are the standard example, and the interface even carries method_has_implemented_ embedding, which inspects whether a method overrode the base’s NotImplementedError stub before routing embedding lookups through it. One checkpoint format is a family of per-layer decisions, not a single switch.

Two lifecycle details reward attention. process_weights_after_loading is the repacking hook — its docstring offers “transpose weights for computation” as the canonical use, and formats with hardware-specific layouts do their rearrangement here rather than in the checkpoint. And the uses_meta_device flag marks methods that create weights on the meta device and quantize layer-wise during loading, “reducing peak memory during loading” — online quantization exists partly as a loading-memory strategy.

The KV-scale mapper is the most surprising member. get_cache_scale_mapper returns a table of regular expressions renaming checkpoint scale tensors — “Deprecated fused kv_scale -> attn.k_scale,” ModelOpt layouts, fused QKV projections, several model-specific spellings — so that, in its own words, “individual model load_weights methods do not need to know about KV-cache scales.” The interface even declares a list of scale suffixes (.q_scale, .k_scale, .v_scale, and zero-point variants) that may appear in a checkpoint without a matching model parameter and should be ignored rather than rejected. The names of scale tensors are a compatibility surface all their own.

SGLang’s base_scheme.py draws the boundary differently: BaseLinearScheme and BaseMoEScheme are separate abstract classes, so the layer-family split is structural rather than per-layer dispatch. Its apply_weights docstring locates the work precisely — “this is where scheme-specific dequant/quant steps/kernels should be applied” — the same dequantize-in-kernel step the third diagram above walked. Both registries, read together, are the chapter’s chain made concrete: format, method, kernel, and device each own one link.

Quality needs a workload-specific gate

Perplexity can detect broad language-model changes but may miss the product behavior that matters. A coding service should test code tasks. A tool-using agent should test tool selection and valid arguments. A long-context service should test retrieval and generation at target lengths.

Measure the unquantized and quantized models with the same prompts, decoding rules, templates, and evaluation. Include calibration-sensitive tasks, rare tokens, structured outputs, and log probabilities if callers depend on them. Log-probability drift deserves first-class status: downstream routers, classifiers, and confidence thresholds consume those numbers, and a format can preserve every task-level metric while shifting the distribution under them.

Do not hide output changes behind a speed average. Report quality alongside latency, throughput, memory, and cost. If a lower-precision model requires more retries or longer outputs to solve the same task, token throughput exaggerates its value — a fifteen-percent throughput win erased by a twenty-percent retry rate is a loss wearing a win’s clothes.

Numerical reproducibility is a separate choice

Greedy decoding returns the same token only when logits remain ordered the same way. Quantization, batch shape, fused reductions, parallel collectives, and attention backend can change rounding. Two close candidates may swap order — and a swap at the top of the distribution is a different response, not a slightly different one.

Reproducibility is an execution contract, not a sampler setting.

flowchart LR
    R["Same request and seed"] --> A["Batch shape A"]
    R --> B["Batch shape B"]
    A --> RA["Reduction order A"]
    B --> RB["Reduction order B"]
    RA --> L["Logits"]
    RB --> L
    L --> C{"Required contract"}
    C -->|Strict| T["Exact tokens"]
    C -->|Analytical| P["Stable probabilities"]
    C -->|Product| E["Task-level equivalence"]

A two-number walk shows how little it takes. Suppose the top two logits are 10.32 and 10.28 under the baseline engine. A different reduction order nudges them to 10.29 and 10.30; greedy decoding now emits the other token, and every subsequent position conditions on it. Under temperature sampling the flip probability per position is tiny, but a 200-token response offers 200 chances: one percent per position compounds to roughly 87 percent of responses differing somewhere — declared figures for the sake of the shape of the argument. This is why “the model got better or worse” is the wrong frame for many quantization comparisons; the right frame is whether response-level differences stay inside what the product treats as equivalent.

Strict batch invariance or reproducibility may require deterministic kernels, fixed reduction order, controlled random state, and restrictions on dynamic batching. Those choices can reduce performance. Decide whether the product needs exact token equality, statistically equivalent sampling, stable log probabilities, or only task-level quality — four postures with very different price tags, and the cheapest one that satisfies the contract is the right one. A/B tests and evaluation pipelines have their own stake: an evaluation that cannot reproduce itself across batch shapes measures the scheduler as much as the model.

A seeded sampler controls one source of variation; it does not make the execution path deterministic. Batch-invariant service also needs operators whose results do not depend on batch composition, deterministic collective and attention implementations, and a stable mapping from requests to random-number streams. The official vLLM batch-invariance guide documents the required execution controls, while SGLang’s deterministic-inference guide shows why kernel and scheduling choices are part of the contract.

Test the contract as a matrix rather than as one repeated command: run the same requests alone and in changing batch shapes, with graph replay on and off, across supported attention backends, and across the intended parallel layout. Record exact-token equality, maximum log-probability drift, and task-level equivalence separately. A product that needs only the last measure should not pay the full throughput cost of the first; an evaluator that compares small logit changes cannot quietly settle for the last.

Run a four-axis evaluation

Choose two candidate quantization strategies and one unquantized baseline. Evaluate them on:

A deployable format must pass both a systems gate and a quality gate.

flowchart TB
    F["Candidate format"] --> M{"Fits memory and has target kernels?"}
    M -->|No| R["Reject for this platform"]
    M -->|Yes| P["Measure workload performance"]
    P --> Q{"Passes product quality and stability?"}
    Q -->|No| R
    Q -->|Yes| D["Deploy for the qualifying tier"]

1. memory: weights, cache capacity, workspaces, and peak allocation;
2. performance: TTFT, ITL, throughput, and goodput over several batch shapes;
3. quality: product tasks plus long-context and structured-output checks;
4. stability: repeated runs, batch changes, and log-probability differences.

Record the exact model artifact, calibration method, engine commit, kernel, device, and command — enough for a colleague to reproduce the number rather than believe it. The winning format is the one that improves the service’s constraint—not the one with the fewest bits in its name.

Worked example: bits do not choose the winner

Compare BF16, weight-only INT4, and FP8 weights with FP8 KV state. INT4 can cut weight storage to roughly one quarter plus scales, but it leaves KV capacity unchanged and may pay dequantization or weak small-batch kernels. FP8 can reduce both weight and cache bytes, which may avoid long-context preemption even when one isolated operation is not faster.

The Atlas planning numbers turn that sketch into arithmetic. Weights are 140 GB in BF16; a 4-way TP shard therefore holds about 33 GiB of them, and INT4 at one quarter plus the 3 percent scale overhead from the tile walk brings a shard to about 8.4 GiB, FP8 to about 16.3 GiB. Chapter 4’s admission budget is 80 GiB per rank minus weights minus a 12 GiB reserve, and each 8,000-token sequence costs 625 MiB of KV per shard in BF16, or about 313 MiB with FP8 state. The budgets then admit, per rank: about 57 sequences on BF16, about 98 on INT4 with BF16 state — the freed weight bytes buy context even though the KV format never changed — and about 169 with FP8 weights and FP8 state. Halving the KV bytes alone doubles capacity per GiB of budget; the weight shrinkage adds more on top.

The ITL side walks just as far. Chapter 4 priced HBM at 3 TB/s, so a batch-1 decode step that reads a 33 GiB BF16 shard spends at least 11 ms on weight traffic alone. An INT4 kernel that sustains full bandwidth would read about 8.4 GiB — under 3 ms — but real 4-bit kernels rarely sustain peak while dequantizing; at 70 percent of peak the step is nearer 4 ms. Either way the direction is decisive if the small-batch kernel is good, which is precisely what “test batch-1 decode kernels” exists to learn: the same checkpoint can deliver the 11-to-4 ms win on one engine and lose most of it on another whose kernel converts poorly at batch 1.

The correct decision begins with the binding constraint. If interactive ITL is the problem, test batch-1 decode kernels. If long documents exhaust memory, measure admitted contexts and preemption. Gate both against product quality, tool-call validity, long-context retrieval, and numerical stability.

Practice: make a deployment decision

Evaluate those three formats on the same Atlas trace. Record weight, KV, workspace, and peak bytes; TTFT, ITL, throughput, and goodput across batch shapes; product-task quality; schema validity; repeated-run and log-probability drift.

Choose a format for interactive and long-document tiers separately, and name the constraint that justifies each choice. Compare your reasoning with Appendix G.

11. Speculative Decoding

Autoregressive generation normally pays for one large-model step per token. If a response contains 200 tokens, the target model runs at least 200 serial steps. That serialization is not an accident — each token conditions on the last — but it is also not sacred: nothing prevents the model from checking several guesses in one pass, because verification of a fixed candidate sequence is parallel while generation of it is not.

Speculative decoding asks whether a cheaper method can guess several future tokens and let the target model verify them in parallel. When the guesses are good and cheap, one target step advances the sequence by more than one token. When they are bad, the engine pays for the guesswork and the verification and advances no faster than before. The whole discipline is in knowing which regime the traffic is in — and in the machinery that keeps a rejected guess from corrupting committed state.

Draft, verify, accept

The classic method uses a small draft model. The draft proposes a run of tokens. The target model evaluates the proposed positions together. An acceptance rule keeps a valid prefix of the proposal and corrects the first rejected position.

Speculative decoding proposes several tokens but commits only verified work.

flowchart LR
    C["Current accepted prefix"] --> D["Draft proposes tokens"]
    D --> V["Target verifies proposal"]
    V --> A{"Accepted prefix length"}
    A --> K["Commit accepted tokens"]
    K --> C
    A --> F["Sample correction at first rejection"]
    F --> C

Where the draft runs is itself a placement decision. On the same device it steals compute from the target between steps; on a separate device it adds a transfer of features or tokens per round; on the CPU it avoids device contention but rarely keeps up with a multi-millisecond step. The draft also owns KV state of its own — it must attend over the same growing context to propose well — so the table’s “draft weights and KV” line is a second cache growing in lockstep with the first, and the memory accounting of the next section has to count it.

draft:   [the] [device] [is] [ready]
target:    ✓      ✓      ✗
result:  accept two tokens, correct the third, discard the rest

With the appropriate rejection-sampling rule, speculation preserves the target model’s output distribution. The intuition behind the rule: a draft token is accepted in proportion to how much the target agrees with the draft, and a rejected position is resampled from the target’s own distribution renormalized over what remains — so the committed stream is statistically indistinguishable from pure target sampling. The foundational speculative decoding paper describes this exact approach. Distribution preservation is the property that makes the technique a serving optimization rather than an approximation: quality is not traded for speed, only latency and throughput are reshaped.

Two invented probabilities make the rule inspectable. Suppose the draft’s top choice is “device” with draft probability 0.8 while the target gives it 0.6: acceptance probability is min(1, 0.6/0.8) = 0.75, so three times in four the target keeps the draft’s guess. On rejection, the token is resampled from the target’s distribution minus the mass already spent on the draft choice, renormalized — which is exactly why the committed stream has the same distribution as if the target had sampled alone, even though the draft chose what to offer. The agreement rate sets both the acceptance probability and how much correction mass remains, which is why drafter-target agreement, not draft quality alone, is what an acceptance metric should surface.

The speedup depends on more than acceptance rate. Drafting consumes compute. Verification uses larger target batches and extra KV slots. Rejected tokens create wasted work. The scheduler and graph runner must support several proposal lengths.

A simple mental model is:

benefit = target steps avoided
cost    = drafting + larger verification + rejection waste + coordination

Speculation helps only when the saved target work exceeds the full cost.

What verification actually costs

The verification batch is the hidden price, and Chapter 4’s arithmetic prices it. A decode step’s arithmetic intensity is roughly the batch size, and the crossover where decode stops being memory-bound sits near intensity 333 for the Atlas hardware class. With three speculative tokens, every decode slot carries 1 + 3 = 4 tokens — the same stride Chapter 9 found in uniform_decode_query_len. At batch 8 the verification step runs at intensity about 32, still far below crossover, so the extra tokens ride along almost free: memory-bound decode was reading the weights once regardless. At batch 128 the ordinary step already runs near intensity 128, and verification quadruples it to about 512 — past crossover, where the target is no longer idle and every speculative token competes with real work. The same proposal length that is free for an interactive tier is a tax under high concurrency.

Memory carries a quieter version of the same bet. Before acceptance is known, each active sequence needs reserved positions for its proposal — lookahead slots the scheduler cannot offer to ordinary decode. At 320 KiB per token, four lookahead positions cost about 1.3 MiB per sequence: trivial per request, but it is capacity that serves no user when acceptance is low, and the graph runner needs buckets covering the padded proposal shapes on top. Speculation’s costs arrive as capacity, scheduling, and shape coverage before they arrive as milliseconds.

Several ways to propose tokens

A separate draft model is not the only source of guesses.

Multi-token prediction heads and EAGLE-style drafters use features from the target model to propose future tokens. The advantage is structural: an EAGLE-style drafter conditions on the target’s own hidden state, so its first guess already benefits from everything the target computed this step, and its drafts correlate with target behavior in a way an independent small model cannot match. Native MTP layers are trained into some model architectures — the head ships with the checkpoint, and its acceptance profile is a property of the model rather than of a separately chosen draft. N-gram and suffix methods search the prompt or recent history for repeated continuations. They add little model compute and work well on repetitive text, but fail when the continuation is novel.

Tree methods propose several branches so the target verifies multiple possible continuations. They may improve the chance of advancing and also enlarge the verification workload — and the enlargement is multiplicative, not additive. A chain of three drafts verifies three positions; a tree with two branches at each of three depths verifies up to seven nodes, and the KV lookahead reservation grows with the node count, not the path length. The attention kernel must mask across the branch structure, so tree verification also narrows kernel compatibility. Trees pay off where single chains stall: text that is locally unpredictable but globally repetitive, where several continuations are plausible and committing to one wastes the round.

At the pinned revisions, vLLM contains draft-model, EAGLE, MTP, n-gram, suffix, DFlash, and dynamic verification paths under vllm/v1/spec_decode. SGLang’s corresponding implementations live under srt/speculative. Support and compatibility vary by model and backend; the source trees illustrate the design space rather than one default recipe.

Guided reading: two proposers, two philosophies

vLLM’s ngram_proposer.py is the zero-model end of the design space. Its load_model method is literally “No model to load”: the proposer searches each request’s own tokens — prompt plus generated history — for the longest suffix that appeared earlier, then proposes the tokens that followed it. The search is a Knuth-Morris-Pratt failure-function scan over the reversed token array, compiled through numba, with the match length capped at max_ngram “to save memory” and a subtle tie-break the comment explains: on equal-length matches it keeps “the earliest position in the original tokens,” preferring the first time history repeated rather than the most recent. The systems details are as instructive as the algorithm. The constructor runs one throwaway proposal to trigger JIT compilation — warm-up discipline, same as Chapter 9’s captures. The thread budget is deliberately tiny: capped at one thread, then divided by TP size “to ensure each tensor parallel rank has some threads since all ranks will run this,” because — the comment continues — “other components like frontend (incl tokenization) and Structured Outputs also use multiple threads.” A CPU-side proposer competes for cores with everything else on the host, and every rank runs it redundantly to stay in lockstep.

SGLang’s adaptive_spec_params.py is the closed-loop end: it treats proposal length as a control variable. DEFAULT_ADAPTIVE_CONFIG assigns each batch size its own candidate set — steps [1, 3, 7] at batch 1, [0, 1, 3] at batch 8, [0, 1] at batch 32, and [0] at batch 64. Read that last entry carefully: by default, speculation disables itself under high concurrency, exactly the regime the intensity arithmetic above predicted. Each slot tracks an EMA of accepted draft length and follows the docstring’s rule — target_steps = clamp(round(ema_accept_len) + 1, min_steps, max_steps) — probing one step beyond observed acceptance, updating only every five batches after ten warm-up batches so the controller does not chase noise. Hysteresis is asymmetric (dropping is easier than rising), a zero-step interval is treated as a probe state that restarts from the smallest positive candidate, and the EMA ceiling “only caps downward — never blocks step-ups, so the system can explore higher steps and let the EMA catch up.” Routing closes the loop with Chapter 9: _route(batch_size) pads the batch to its CUDA-graph size and picks the nearest configured slot, so the controller only ever selects step counts the graph buckets can execute. And adaptive_unsupported_reason enumerates what the controller cannot coexist with — DP attention (“adaptive tier decisions are not synchronized across DP ranks”), two-batch overlap (“adaptive state swap would discard the TboAttnBackend wrapper”) — a list that reads like a map of Part III. A runtime-tuned knob is still a knob with preconditions. Read together, the two files bracket the design space: one buys proposals with CPU cycles and an old algorithm, the other with a control loop over the serving systems this whole part has built.

Proposal length should adapt

A fixed proposal length is easy to graph and schedule. It is wasteful when acceptance changes.

A speculation round is a strip of drafts with a rollback point at the first rejection.

flowchart LR
    P["Committed prefix"] --> T1["Draft t+1"] --> T2["Draft t+2"] --> T3["Draft t+3"] --> V{"Verify all positions in one target step"}
    V -->|"all accepted"| C["Commit through t+3"]
    V -->|"reject at t+2"| B["Commit through t+1;<br/>t+2 rolls back and resamples"]
    C --> P
    B --> P

At low batch size, target decode may be memory-bound, making a larger verification batch relatively cheap. Under high concurrency, the target is already efficient and draft work competes with useful requests. Some prompts are predictable; others are not. Acceptance also changes during one response — code and boilerplate accept well; the sentence that introduces a new idea accepts poorly.

An adaptive policy can use recent acceptance, batch size, draft confidence, or an estimated cost model to choose the number of proposed tokens—or disable speculation entirely. The policy must be stable enough to avoid frequent graph misses and schedule churn, which is why the controller above smooths with an EMA, hysteresis, and an update interval instead of reacting to every batch.

Measure accepted tokens per target step, not acceptance percentage alone. A method accepting 80 percent of two proposals advances less than one accepting 60 percent of eight if their costs are comparable.

The expectation arithmetic shows why proposal length hits diminishing returns fast. If each draft position independently survives with probability p, a chain of k proposals advances 1 + p + p² + … + pᵏ tokens on average — the 1 is the correction token every round earns. At p = 0.8 and k = 3 that is 1 + 0.8 + 0.64 + 0.51 ≈ 2.95 tokens per round. Stretching to k = 6, each new position buys less than the last — the fourth draft is worth p⁴ ≈ 0.41 tokens, the fifth 0.33, the sixth 0.26 — while every one of them costs a full verification slot and lookahead reservation. The geometric tail means long chains only pay when acceptance is very high: at p = 0.95 the same stretch from three to six adds about 2.3 tokens per round, which is why predictable traffic tolerates aggressive proposal lengths that would bankrupt unpredictable traffic.

Speculation changes memory and scheduling

The target needs temporary positions for proposed tokens. Accepted positions become normal sequence state; rejected positions must not remain visible. Chunked prefill creates boundaries where a drafter may need additional lookahead. Asynchronous scheduling can prepare the next step before acceptance is known, so it must reserve conservatively and repair state afterward — Chapter 6’s overlap machinery now settles bets, not just samples.

Memory pressure turns the lookahead reservation into a preemption input, using Chapter 6’s distinctions exactly. A preempted sequence that is swapped moves its reserved lookahead blocks to the host too — paying transfer cost on state that may be discarded unverified when the sequence returns. A preempted sequence that is recomputed pays 0.06 ms per token for its whole accepted context under Atlas’s constants — including tokens that speculation produced nearly for free, which are now ordinary committed history that must be earned back the slow way. An admission controller that counts only committed context will find sequences heavier than they look; the lookahead slots are part of the sequence’s true footprint.

Parallel execution adds synchronization. Every rank must agree on accepted lengths and state mappings; a single rank that kept a rejected token would fork its KV cache from its peers’ and corrupt every subsequent collective. Disaggregated decode adds another question: where does the drafter run, and which state crosses the network?

This is why speculative decoding is a serving algorithm, not a wrapper around model calls.

Constraints change verification

A draft token forbidden by the request’s grammar cannot be accepted. Verification must advance model state and parser state in lockstep, then roll both back at the first rejected position. The ordinary and speculative paths therefore need the same legality rule for every proposed token.

Chapter 22 owns grammar compilation, bitmask construction, reasoning and tool parsers, and structured streaming. The interaction to retain here is narrower: constraint complexity changes verification cost and proposal acceptance, so speculation must be benchmarked with the same schemas production traffic uses.

Worked example: acceptance is not speedup

Ordinary target decode costs 8 ms per token. A speculative step spends 3 ms on drafting and 9 ms on verification. At 3.2 accepted tokens per step, it costs 3.75 ms per accepted token. At 1.3 accepted tokens, it costs 9.23 ms and loses before accounting for extra memory.

Walk the arithmetic once more to find the hinge. The speculative round always pays 3 + 9 = 12 ms and advances by however many tokens survive verification, so its per-token cost is 12 divided by accepted tokens. Setting that equal to the 8 ms baseline gives the break-even: 12 / a = 8 means a = 1.5 accepted tokens per round. Above 1.5, speculation wins; below, it loses — and the distance matters as much as the side. At 3.2 accepted the win is 2.1×, worth real money; at 1.6 it wins by six percent, which a small regression in draft latency or one unlucky traffic shift erases. A deployment that tracks only “acceptance rate” sees 75 percent in both cases and cannot tell them apart; accepted tokens per round — the metric this chapter has argued for — is the number that crosses the line.

That produces an online decision rule: estimate accepted tokens and compare draft plus verification plus capacity cost with ordinary target work. The capacity cost has a walkable floor: a 1B-parameter draft model held in BF16 costs 2 GB, which against Chapter 4’s 35 GiB admission budget is about three fewer 8,000-token sequences per rank — 57 becomes 54, a five-percent concurrency give before the first request arrives. A speculation win must clear that bar too, not just the per-step arithmetic. Disable speculation for short remaining outputs, low recent acceptance, memory pressure, or graph-incompatible shapes. Acceptance rate remains an input to the decision, not the result.

Practice: find and explain the losing regime

Benchmark ordinary decode and two proposal strategies on predictable, unpredictable, short-output, and high-concurrency traffic. Record draft and verification time, accepted tokens, extra memory, graph dispatch, TTFT, ITL, and output-distribution checks.

Use the numbers above to derive the break-even acceptance level, then add one capacity penalty observed at concurrency. Write a rule for turning speculation off. The worked calculation is in Appendix G.

We have now followed a request through one engine, from allocation to kernels and decoding. Part III expands the same ideas across multiple accelerators and machines.

12. Adapter Serving and Multi-Tenant Customization

A single base model can serve many customers, but not all customers want the same model. A legal team needs answers tuned for contract language. A medical group needs clinical tone and terminology. A retailer needs product-catalog fluency. Full fine-tuning would give each customer a private 140 GB checkpoint, and the fleet would need one replica per customer. Low-rank adapters offer a different deal: each customer gets a small weight delta – a few hundred megabytes – that modifies the shared base model at inference time. One replica serves many customers from one copy of the base weights.

The economics are attractive enough that production deployments routinely serve dozens to hundreds of adapters on a shared fleet. The engineering is less simple than the pitch. Adapter weights compete with KV state for the same HBM. Scheduling must decide which adapters share a batch. The router must know which replicas hold which adapters. Prefix sharing, speculative decoding, CUDA-graph capture, and quantization all interact with the adapter dimension in ways that are invisible until they produce wrong answers or surprise latency. This chapter treats adapter serving as the scheduling, memory, and routing problem it becomes at scale.

What an adapter adds to a forward pass

A LoRA adapter decomposes a weight update into two low-rank matrices. Where the base model applies a weight matrix W of shape [d, d], the adapted model computes W*x + B*A*x, where A is [r, d] and B is [d, r] for rank r. Typical serving ranks are 8 to 64; rank 16 on Atlas’s hidden size of 8,192 gives A and B matrices of 8192 x 16 each. At BF16, one such pair costs 2 x 8192 x 16 x 2 = 512 KiB. Applied to attention projections (Q, K, V, O) across 80 layers, a rank-16 adapter totals about 4 x 512 KiB x 80 = 160 MiB. Extending to MLP layers (gate, up, down projections) roughly doubles that to 320 MiB – still less than 0.25 percent of the 140 GB base model.

Adapter weights are thin overlays on the shared base model.

flowchart LR
    B["Base weights (read-only, 140 GB)"] --> F["Forward pass"]
    A1["Adapter A weights (~160 MiB)"] --> F
    A2["Adapter B weights (~160 MiB)"] --> F
    F --> O["Per-request output"]

The compute is proportionally small. Each adapted layer adds two matrix multiplications of rank r against the batch. At rank 16 the extra FLOPs per token per layer are 2 x 2 x 8192 x 16 = 524,288 – about 0.4 percent of the base layer’s 2 x 8192 x 8192 = 134 million FLOPs for a single projection. The adapter’s arithmetic is noise against the base model’s work. The systems cost is not in compute; it is in memory, identity, and placement.

Memory accounting for a multi-adapter fleet

Base weights are shared; adapter weights add up

The 140 GB base model is loaded once per replica and never modified during serving. Every adapter request reads the same base weights – without this sharing, serving N customers would require N copies. But each adapter is an additional allocation:

50 adapters x 160 MiB (rank 16, attention only) = 8 GB
50 adapters x 320 MiB (rank 16, attention + MLP)  = 16 GB

Sixteen gigabytes is not fatal on an 80 GB device, but it is not free either. Chapter 7 showed that Atlas’s KV budget is roughly 35 GiB after base-model weights, activation buffers, and graph pools. Sixteen gigabytes of adapter weights cuts that budget nearly in half, reducing maximum concurrency from about 57 resident 8,000-token conversations to about 30. The adapter memory competes with KV cache for the same HBM, and the competition has a clear loser: every gigabyte spent on adapter weights is a gigabyte that cannot hold KV state, which means fewer concurrent sequences, more preemption, or both.

Tiered storage: hot, warm, cold

Not all adapters are equally active. Traffic typically follows a power law: a handful of popular adapters serve most requests while a long tail sees occasional use. A tiering strategy matches storage cost to access frequency:

A rank-16 adapter at 160 MiB transfers over PCIe Gen5 x16 (64 GB/s) in about 2.5 ms. In practice, the transfer is not the whole cost: the engine must allocate destination buffers, update pointer tables, and potentially invalidate CUDA graphs. Measured cold-load times for typical adapters run 20 to 100 ms from host memory – the number Chapter 17 priced at 800 ms includes disk-resident adapters on a cold path with no pipelining.

S-LoRA’s unified paging

The S-LoRA paper observed that adapter pages and KV pages share the same management problem: both are variable-size allocations that arrive and depart with requests, both benefit from paging to avoid fragmentation, and both compete for the same physical memory. S-LoRA’s Unified Paging allocates adapter weight pages from the same pool as KV pages, using the same block table machinery Chapter 7 described. The Punica kernel executes batched LoRA operations where each sequence in the batch may use a different adapter, gathering the correct A and B matrices through indirection rather than requiring all sequences to share one adapter.

This unification has a scheduling consequence. The block allocator now manages two kinds of tenants – KV blocks and adapter blocks – and admission must account for both. A request that arrives for a cold adapter needs adapter blocks allocated before its first prefill token, and those blocks reduce the KV capacity available to every other request in the batch.

vLLM’s pre-allocated LoRA buffers

vLLM takes a different approach at the pinned revision. The --max-loras flag sets the maximum number of adapters active in one batch, and --max-lora-rank sets the maximum rank. The engine pre-allocates fixed buffers sized to hold that many adapters at that rank, reserving the memory at startup rather than paging it dynamically. The trade-off is fragmentation for predictability: the buffers are always allocated whether or not they are full, but the engine never needs to page adapter weights during a step.

At the pinned commit, vLLM’s LoRA implementation lives under vllm/lora. The LoRAManager coordinates adapter loading, and the punica_wrapper contains the batched LoRA kernels that apply different adapters to different sequences in a single matrix multiplication.

SGLang’s adapter support at the pinned revision lives under srt/lora.

Scheduling with adapter awareness

Batching across adapters

Multi-adapter scheduling groups requests by active adapter within a mixed batch.

flowchart TB
    Q["Waiting requests with adapter tags"] --> S["Adapter-aware scheduler"]
    S --> G1["Group: Adapter A requests"]
    S --> G2["Group: Adapter B requests"]
    S --> G3["Group: base-only requests"]
    G1 --> M["Mixed batch with per-request adapter pointers"]
    G2 --> M
    G3 --> M
    M --> E["Execute one step"]

The simplest approach batches all requests together regardless of adapter. The base-model forward pass runs once for the full batch, and each sequence’s adapter contribution is added through gathered low-rank products. This is what the Punica kernel enables: the base matmul is one operation; the adapter matmuls are a second, indexed operation that reads different A and B matrices per sequence. The batch sees one base-weight read and many small adapter reads.

The cost model is straightforward. The base-weight read dominates at 140 GB per step; adapter weights add at most a few hundred megabytes of extra reads. Mixed-adapter batching is therefore nearly free in steady state, provided all adapters are already resident.

Switching cost: I/O, not compute

The cost appears when an adapter is not resident. Loading a warm adapter from host memory takes tens of milliseconds. Loading a cold adapter from disk takes hundreds. During that time, either the request waits (adding to TTFT) or the engine stalls (adding to every request’s ITL).

The switching cost is I/O-bound, not compute-bound. A rank-16 adapter’s 160 MiB is a memory transfer, not a matrix multiplication. This means it can overlap with compute: while the GPU executes early layers, the engine can transfer a cold adapter’s weights for later layers. Layer 40’s adapter weights are not needed until the forward pass reaches layer 40, so they can arrive while layers 0 through 39 execute. This is the same overlap principle Chapter 15 applied to KV transfer in disaggregated serving, and the opportunity is better here because adapter weights are smaller than multi-gigabyte KV images.

Adapter-aware grouping

When adapter-loading cost is non-trivial, the scheduler can reduce it by grouping requests that share an adapter. If ten requests for adapter A and two for adapter B are waiting, scheduling all ten A-requests together avoids loading adapter B until the next step. This is adapter-affinity scheduling: prefer to fill the batch with requests that share already-resident adapters.

The risk is starvation. If adapter A is popular and adapter B is rare, strict affinity scheduling can delay B-requests indefinitely. Fair scheduling across adapters requires the same discipline Chapter 6 applied to priority classes: bound the maximum wait time, reserve minimum batch slots for underserved adapters, or use weighted round-robin across adapter groups. The popularity power law makes this concrete: if 80 percent of requests use the top 5 adapters, the remaining 45 adapters share 20 percent of batch capacity and need protection from indefinite deferral.

Routing and adapter placement

Each replica holds a different adapter set

Adapter placement turns routing into a three-term cost.

flowchart TB
    R["Request with adapter tag"] --> C["Candidate replica"]
    C --> QT["Estimate queue time"]
    C --> PT["Estimate missing-prefix compute"]
    C --> AT["Estimate adapter-load time"]
    QT --> SC["Combined routing score"]
    PT --> SC
    AT --> SC
    SC --> D["Choose destination"]

In a fleet of replicas, not every replica needs every adapter. If the hot set is 5 adapters covering 80 percent of traffic, those 5 should be resident on every replica. The remaining 45 can be distributed: some replicas hold adapters 6 through 25, others hold 26 through 50. A request for adapter 37 routes to a replica that already has it, avoiding the cold-load penalty.

This is the adapter term in Chapter 17’s routing score:

cost(R) = queue(R) + missing_tokens(R) x 0.06 ms
        + adapter_load(R) + risk(R)

The adapter_load(R) term is zero when the target replica already holds the requested adapter and nonzero – potentially hundreds of milliseconds – when it does not. Chapter 17 priced the loss: sending a request to a replica without its adapter costs 800 ms of foreground load time on first use, which dominates both the queue and missing-prefix terms in most scenarios.

Power-law traffic and placement strategy

Adapter popularity follows a power law. Zipf with exponent near 1.0 is a reasonable model: the most popular adapter sees roughly 50 times the traffic of the median adapter. The placement strategy follows:

• Universal hot set. The top few adapters are resident everywhere. Their per-replica memory cost is small (5 adapters at 160 MiB = 800 MiB), and their traffic share justifies the HBM.

• Partitioned warm set. The next tier is distributed across replicas. Each replica holds a subset, and the router sends requests to replicas that have the right adapter. The partition should be rebalanced as popularity shifts.

• On-demand cold tail. Rarely used adapters stay on disk or in a shared store. A request for a cold adapter pays the full load penalty, but it happens infrequently enough that the fleet-level impact is small.

Why naive round-robin fails

Round-robin routing ignores the adapter dimension entirely. With 50 adapters and 8 replicas, every adapter eventually receives traffic on every replica. Each replica must eventually load all 50 adapters, spending 50 x 160 MiB = 8 GB of HBM on adapter weights – regardless of whether most of those adapters see only one request per hour on that replica. Worse, each cold-load event adds tens to hundreds of milliseconds to the affected request’s TTFT, and the cold loads are scattered unpredictably across the fleet.

Adapter-aware routing concentrates each adapter’s traffic on a small number of replicas, keeping the per-replica footprint proportional to actual use. The router needs one additional piece of telemetry: each replica’s adapter inventory.

Interactions with other engine mechanisms

KV cache identity

Chapter 7 established that cached KV state depends on the full identity: model version, tokenizer, token IDs, positions, and adapter. Two requests with identical prompts but different adapters produce different KV state, because the adapter modifies the attention projections that generated the keys and values. Sharing cached prefix blocks across adapters is invalid – the same text under a different adapter is a different model, and reuse produces silently wrong outputs.

This means prefix sharing in a multi-adapter deployment is scoped to requests that share both the same prompt prefix and the same adapter. The cache hit rate drops as the adapter count grows: with 50 adapters, a system prompt cached under adapter A benefits only the fraction of traffic using adapter A. The total cache value of a prefix is its per-adapter hit rate times the number of adapters that share it – and for adapter-specific prefixes, that number is one.

Speculative decoding

A draft model proposes tokens that the target model verifies. When the target model uses an adapter, the draft must produce proposals consistent with the adapted model’s distribution. A base-model draft proposing tokens for an adapted target will have systematically lower acceptance rates wherever the adapter has shifted the distribution, reducing the speedup or eliminating it entirely.

The options are: apply the same adapter to the draft model (doubling the adapter’s memory footprint), use a draft model fine-tuned alongside the adapter (requiring one draft per adapter, which rarely exists), or accept lower acceptance and let the adaptive controller from Chapter 11 reduce or disable speculation for adapter traffic. The third is simplest in practice.

Quantization

Adapter weights may use a different numerical format from the base model. The base model might be quantized to INT4 for memory savings while the adapter weights remain in BF16 for quality. The kernel must handle mixed-precision arithmetic: dequantize the base weights, add the BF16 adapter contribution, and accumulate in a wide type. This is the same concern Chapter 10 raised for mixed quantization generally, but adapters make it per-request: one batch may mix INT4-base-plus-BF16-adapter sequences with INT4-base-only sequences.

CUDA graphs

Chapter 9 showed that CUDA-graph capture keys include adapter state. At the pinned vLLM revision, graph keys are the cross product of batch sizes and LoRA counts: product(cudagraph_capture_sizes, lora_cases). With 5 batch sizes and 4 LoRA cases, the engine captures 20 graphs. Each capture takes about 0.7 seconds, so warm-up is 14 seconds – tolerable for startup, but the graph pool grows with the product.

Adapter switching can invalidate a captured graph. A graph captured with adapter A’s weight pointers baked in will silently apply adapter A’s weights to every sequence, regardless of which adapter the sequence actually uses. Dynamic pointer resolution – reading adapter addresses from a buffer rather than baking them into the graph – avoids this but requires the graph to include the indirection. Chapter 9’s observation applies directly: a captured graph with baked adapter weights silently serves the wrong model.

Worked example: Atlas adds fifty customer adapters

Price the memory

Atlas base model: 140 GB in BF16. Each rank-16 adapter across the four attention projections (Q, K, V, O) in 80 layers:

Per adapter:
  4 projections x 80 layers x 2 matrices x 8192 x 16 x 2 bytes
  = 4 x 80 x 2 x 8192 x 16 x 2
  = 4 x 80 x 512 KiB
  = 160 MiB

50 adapters:
  50 x 160 MiB = 8 GB

Total weight footprint: 140 + 8 = 148 GB. On a 4-way tensor-parallel deployment across 80 GB devices, each device holds 35 GB of base weights and 2 GB of adapter weights, leaving about 43 GB for KV cache, activations, and graph pools. Compared to the adapter-free 45 GB, the adapter overhead is about 4.5 percent of device memory – modest for 50 customers.

If adapters also cover MLP projections (gate, up, down), each adapter doubles to about 320 MiB, and 50 adapters cost 16 GB total or 4 GB per device. The KV budget drops from 45 to 41 GB per device, a loss of roughly 5 additional 8,000-token conversations per device.

Adapter-aware routing saves cold-load time

Assume Zipf-distributed traffic with exponent 1.0 across 50 adapters, served by 8 replicas.

Naive round-robin. Each replica receives traffic for all 50 adapters. Assuming each adapter’s first arrival on a replica costs 50 ms to load from host memory, the fleet pays 8 replicas x 50 adapters x 50 ms = 20,000 ms of cumulative cold-load time during warm-up. More importantly, 400 individual requests (one per adapter per replica) each suffer an extra 50 ms added to their TTFT – potentially breaching the 600 ms target on a request that was otherwise on budget.

Adapter-aware routing. Partition the 50 adapters across replicas: the top 5 adapters (covering roughly 45 percent of traffic under Zipf-1.0) are resident everywhere. The remaining 45 are distributed in groups of about 6 per replica. Each replica holds 5 + 6 = 11 adapters, totaling 11 x 160 MiB = 1.76 GB per replica instead of 8 GB. Cold-load events drop from 400 to 8 x 11 = 88 during warm-up, and the router avoids cold loads in steady state by directing each request to a replica that already holds its adapter.

The steady-state benefit compounds. Under round-robin, adapter 50 (the least popular) might arrive at each replica once per hour, and if eviction has reclaimed its slot, every arrival pays a cold load. Under adapter-aware routing, adapter 50 lives on one replica, sees all its traffic there, and stays warm as long as it receives any traffic at all.

How the routing score changes

Take a concrete request for adapter 37, which is in the cold tail:

Without the adapter term, R1 wins at 0 ms. With it, R1 costs 50 ms and R2 costs 130 ms. R0 at 200 ms loses either way in this snapshot – but if R0’s queue clears before the adapter load on R1 completes, R0 delivers the first token sooner. The adapter term changes the winner and prevents the fleet from scattering cold loads across replicas that will never see a second request for that adapter.

Practice: simulate adapter-aware vs. adapter-blind routing

Generate a synthetic trace: 50 adapters with Zipf-distributed popularity (exponent 1.0), 8 replicas, 1,000 requests arriving at Poisson intervals. Each adapter is 160 MiB. Cold load from host memory costs 50 ms. Each replica starts with no adapters loaded.

Simulate two routing strategies:

1. Round-robin (adapter-blind): requests cycle through replicas in order, ignoring adapter state.

2. Adapter-aware: the top 5 adapters are pre-loaded on all replicas. Remaining adapters are assigned to replicas by hashing the adapter ID. The router sends each request to a replica in the adapter’s assigned set, breaking ties by shortest queue.

Measure:

• Total cold-load events across the fleet.
• Number of requests with TTFT exceeding 600 ms (assuming base TTFT is 400 ms, so a 50 ms cold load is safe but a second stacked load is not).
• Per-replica adapter memory high-water mark.
• 99th-percentile TTFT for each strategy.

The worked calculation is in Appendix G.

Adapter serving is a memory, scheduling, and routing problem with a distinctive shape: the weight overhead per adapter is small, but the interaction with every other mechanism in Part II is not. Adapters change what prefix sharing means, what CUDA-graph keys contain, what the routing score must include, and how much HBM the KV cache actually gets. The next part extends these single-engine concerns across multiple accelerators and machines, where adapter placement becomes a distributed-scheduling problem.

Part III — Scaling Across Accelerators

What changes when computation and state cross device and machine boundaries: partitioning a model across ranks, routing tokens to experts, moving KV state between pools, keeping reusable prefixes alive beyond one accelerator, and the control plane that places work whose best location keeps changing.

Chapters 13–17

13. Parallelism as Data Movement

A model no longer fits on one GPU. The obvious response is to add another GPU. The difficult question is what to split.

You can split weights, layers, tokens, experts, attention heads, sequence positions, or complete requests. Each choice reduces one device’s work or memory by moving something between devices — and every byte moved is paid for on a link with its own latency and bandwidth. Parallelism is therefore best understood as a data-movement plan: pick what moves, how often it moves, and which fabric it moves over. Two plans that partition the same model can differ by an order of magnitude in served latency, because they differ in what crosses which wire.

Start with replication

If the model fits on one device, the simplest scale-out design is replication. Each replica holds the complete model and serves independent requests. This is data parallelism in its inference form.

Parallel dimensions split different objects and create different traffic.

flowchart TB
    M["Model and request work"] --> D["Data parallel: replicas"]
    M --> T["Tensor parallel: layer tensors"]
    M --> P["Pipeline parallel: layer stages"]
    M --> C["Context parallel: sequence positions"]
    M --> E["Expert parallel: experts"]

Replication adds capacity without putting a collective on the critical path of one request. It also duplicates weights and fragments warm state across replicas. The state fragmentation is subtler than the memory duplication: each replica builds its own prefix cache (Chapter 7), so two replicas serving the same popular document each pay to warm it, and a router that bounces a user between replicas silently discards that investment. A router must decide where requests go, and its decision quality becomes part of the cache design.

The router also inherits a queuing problem. Two replicas behind one address see arrival bursts that neither controls; a naive round-robin can hand three long prefills to one while the other sits idle, and the queued request’s TTFT absorbs the difference even though the fleet had capacity. Least-loaded routing fixes the queue and breaks the cache affinity; prefix-affinity routing fixes the cache and re-creates the queue. Production routers end up weighting both signals — load first, affinity as a tiebreaker — because a cache hit worth 350 ms cannot repay a queue worth 900. The exact numbers are workload property, but the tension is structural: Chapter 7 made state valuable, so placement now trades two goods against each other. For latency-sensitive models that fit, replication is the baseline against which more complicated plans should be justified.

Tensor parallelism splits a layer

Tensor parallelism divides the matrices inside a layer across ranks. A common transformer plan shards one linear operation by output dimension and another by input dimension. Partial results are combined with an all-reduce or reduce-scatter.

The benefit is immediate: each rank stores and multiplies only a shard. The cost is also immediate: ranks communicate at layer frequency. Fast links and large matrix shapes can make the exchange worthwhile. Small decode batches or slow cross-node links can make synchronization dominate.

The foundational Megatron-LM paper explains intra-layer tensor parallelism for transformer models. Its training context is different, but the partition and collective reasoning carries into inference.

Tensor parallelism can lower single-request latency when the communication is cheaper than the removed computation. It should not be a reflexive default.

What TP does to attention heads

Tensor parallelism also decides where attention heads live, and the Atlas constants show how sharp that decision is. The model has 8 KV heads of dimension 128; its 320 KiB per token of KV state is exactly 8 heads × 128 × 2 (K and V) × 2 bytes × 80 layers. At TP4, each rank owns 2 of the 8 KV heads — 80 KiB per token, the 625 MiB per 8,000-token sequence that Chapter 3 counted. The division is clean because 4 divides 8. At TP8, each rank would own exactly one head; at TP16, there are no heads left, and implementations replicate heads across ranks — spending memory on copies that compute nothing new. Grouped-query-attention models therefore carry a practical ceiling: tensor width beyond the KV-head count buys no attention memory, only duplication. When a sizing conversation reaches “why not TP8 everywhere,” the head count is the first number to check.

Two ways to cut attention state

Tensor and context parallelism can each hold attention memory down to the same number by splitting different objects — and their traffic differs in instructive ways. Splitting heads (TP): each rank keeps 2 heads × all 8,000 positions = 625 MiB, and attention needs no cross-rank communication at all, because heads are independent — but the rank is now locked into the tensor group for every layer, MLP included. Splitting positions (decode-context): each rank keeps all 8 heads × 2,000 positions = 625 MiB, attention runs locally over its stripe, and the new query — under a kilobyte — must reach every stripe, with partial results reduced back each step. Same memory, same per-rank attention compute; different coupling. TP-attention couples the attention dimension to the whole layer’s tensor width; context parallelism couples it to a per-step collective instead. Deployments choose based on which coupling their other dimensions tolerate — and on whether their attention backend implements the stripe-and-reduce path at all, which is Chapter 8’s registry question wearing a parallelism costume.

Ring attention: the long-context variant

Prefill at extreme context stresses the position-split scheme differently, because every query attends to every key — a rank holding the middle stripe of a 131,072-token context needs all of it, not just its neighbors. Ring attention is the standard resolution: split positions across ranks, then rotate the key-value stripes around a ring so that each rank computes its queries against every stripe exactly once while activations stay home. The cost is pure data movement, and Appendix A prices it. Per rank, Atlas KV for 131,072 tokens is 80 KiB × 131072 = 10 GiB; moving it in segments of 8,192 tokens means each hop carries 0.625 GiB, and with four ranks each rank receives three hops — roughly 3 × (20 µs + 0.625 GiB / 450 GB/s) ≈ 3 × 1.5 ms ≈ 4.5 ms per layer of exposed transfer if nothing overlaps with compute. Against a prefill step’s large matrix work, overlap hides most of that; what remains scales linearly in context length and inversely in ring size, which is why long-context serving treats inter-rank bandwidth as the binding resource, and why doubling context without widening the ring doubles the attention-communication tax per token. The scheme composes with the rest of this chapter: heads can still be tensor-sharded within a rank, and the partial softmax statistics each stripe produces must combine exactly as the decode-context path above requires.

Pricing one all-reduce

Appendix A’s transfer model makes the cost concrete: a transfer of S bytes takes transfer time = a + S / b, where a is startup latency and b is sustained bandwidth. Decode collectives live almost entirely in the a term, and that fact shapes everything. Take Atlas’s width: hidden size 8,192 in BF16 is a 16 KiB activation per token. At decode batch 8, one rank’s contribution to a collective is about 128 KiB; assume intra-island figures of a = 20 µs startup and b = 450 GB/s for a ring-style reduce, and the transfer term is under half a microsecond while the whole collective costs roughly its startup. The message is small; the round trip is not negotiable.

Now count collectives. A transformer layer typically needs two — one after attention, one after the MLP block — so a TP4 Atlas step performs about 160 collectives at 80 layers. At 20 µs each, that is roughly 3.2 ms of synchronization per step, against a weight-read term of about 11 ms per rank (a 35 GB shard over 3 TB/s). Communication is nearly a third of the step even though the messages are tiny. This is why tensor parallelism wants the fastest fabric available and why it stops helping when batch growth pushes decode past the memory-bound knee: once the target is compute-limited, both the compute it saves and the collectives it adds scale together, and the S term starts mattering too. Prefill flips the balance — activations there are megabytes per token-batch, so S / b dominates and wide groups pay real bytes as well as real latency.

Pipeline parallelism splits layers

Pipeline parallelism assigns consecutive layer ranges to different stages. An activation moves from one stage to the next. Each stage stores only its layers, so pipeline parallelism solves model capacity without a collective inside every layer — the communication drops from twice per layer to once per stage boundary, and it crosses whatever link separates the stages.

The pipeline must remain occupied. If only one microbatch is present, later stages wait while the first stage begins and earlier stages wait after their work moves on. These empty periods are pipeline bubbles.

Serving makes scheduling difficult because sequences enter and leave dynamically. Chunked prefills can provide pipeline work, while short decode steps may expose bubbles. Pipeline parallelism is attractive across links where one activation transfer per stage is cheaper than repeated tensor collectives, but it needs enough concurrent work.

Bubble arithmetic and what fills it

The bubble has an arithmetic, and it is brutal at serving batch sizes. With s stages and only one unit of work in flight, exactly s − 1 stages idle during any step: at depth 2 the best steady-state utilization is half the fleet doing nothing. The classic training remedy is many microbatches in flight — utilization approaches m / (m + s − 1) for m concurrent units — but inference cannot manufacture microbatches on demand; it has requests, and their arrival is the workload’s choice, not the scheduler’s.

What serving can do is keep some stream flowing through the empty slots. A chunked prefill supplies stage-filling work while decodes run — Chapter 6’s chunk ceiling exists partly so those chunks stay schedulable alongside decodes. Multiple pipeline-parallel replicas can be staggered so one fills while another drains. And the bubble cost shrinks relative to stage time when stages are long: deep pipelines amortize the boundary crossing over more layers per stage, which is another way of saying pipeline parallelism suits very large models whose stages are inherently busy. None of these tricks make the bubble vanish; they decide whether the fleet’s idle fraction is five percent or fifty.

Splitting sequence positions

Long contexts can make attention state or computation too large for one rank. Context or sequence parallelism divides token positions across devices. Each rank computes a portion of attention and the partial results are combined.

Ring-style attention passes key/value regions around ranks. Ulysses-style methods exchange tensor dimensions so each rank can compute local attention. The two move comparable bytes in a step — both traffic scales with context size, not batch size — but the patterns differ in ways links care about: ring streams stripes neighbor-to-neighbor, tolerating slower fabrics but paying hop-by-hop latency, while Ulysses all-gathers head slices so every rank ends up holding what it needs at once, preferring fast fabric and paying for it in one burst. Decode-context parallelism can stripe the stored context across ranks while a small number of new queries attend to all shards — at Atlas’s 320 KiB per token, striping an 8,000-token sequence over four ranks holds 625 MiB per rank instead of 2.44 GiB, the same arithmetic Chapter 4 used to justify the shard budget.

These methods trade memory capacity and attention compute for communication. They become attractive when context state, not weights, is the limiting resource — long-context services where Chapter 7’s eviction pressure, not weight fit, is the daily constraint.

Expert and attention parallelism

MoE models allow experts to be distributed independently from the attention layers. Expert parallelism places different experts on different ranks and moves token representations to their selected owners.

Its traffic profile differs from every other dimension in one respect: the destination is decided at runtime by the router, not by a fixed partition. Each token’s representation must reach its top-k experts’ ranks — an all-to-all whose message count scales with tokens × k and whose completion time is set by the slowest participant. A hot expert that attracts more than its share turns one rank into a straggler for the whole layer; balancing expert load is therefore part of the parallelism plan, not just a modeling concern, and Chapter 14 walks what it costs.

Some deployments replicate or data-parallelize attention while expert layers span a larger group. This is often called attention data parallelism. It avoids tensor collectives in attention and uses the expert all-to-all as the main cross-replica exchange.

The model no longer has one parallel size. It has a mesh of dimensions — and the expert dimension’s traffic is the most irregular of them all, which is why Chapter 14 gives it a chapter of its own.

Compose a rank mesh

Suppose a deployment has 64 GPUs and chooses:

A rank mesh must be mapped onto the physical fabric.

flowchart LR
    A["Logical TP group 0"] --> I0["Fast-link island 0"]
    B["Logical TP group 1"] --> I1["Fast-link island 1"]
    I0 --> N["Inter-node network"]
    I1 --> N
    N --> P["Pipeline or replica traffic"]

data parallel = 4
tensor parallel = 2
expert parallel = 8

The product is 64 only if those axes are independent in the implementation. Some systems define expert parallel within or across data-parallel groups; others couple sizes. Pipeline and context axes add further constraints.

Write down what each coordinate means. A rank might be identified as:

(replica 2, tensor shard 1, expert shard 6)

Then list the communication groups. Which ranks participate in layer all-reduces? Which participate in expert dispatch? Which share a KV partition? This exercise catches configurations that multiply cleanly but communicate poorly.

Both source snapshots centralize group construction and rank state: vLLM in parallel_state.py and SGLang in parallel_state.py. Reading these files is often the fastest way to learn what a framework’s parallel-size arguments actually compose.

Guided reading: the mesh is a reshape

vLLM’s initialize_model_parallel turns the composition question into linear algebra you can read. The heart of the function is one line:

all_ranks = torch.arange(world_size).reshape(
    -1,
    data_parallel_size,
    pipeline_model_parallel_size,
    prefill_context_model_parallel_size,
    tensor_model_parallel_size,
)

Every rank coordinate system this chapter described is that reshape. Each parallel group is then extracted by transposing its axis to the end and slicing — the comment spells out the recipe (“transpose that dimension to the last dimension, then reshape to 2D, then unbind”), and the getters around it (get_tp_group, get_pp_group, get_dp_group, get_ep_group, get_dcp_group, get_pcp_group) hand back one coordinator per axis. A deployment “composes” sizes exactly when this reshape succeeds without coupling axes it wanted separate.

The docstring works the same example this chapter asked you to try. Eight GPUs with tensor size 2 and pipeline size 4 produce four TP groups — [g0,g1], [g2,g3], [g4,g5], [g6,g7] — and two PP groups [g0,g2,g4,g6], [g1,g3,g5,g7], followed immediately by the topology warning that matters: “for efficiency, the caller should make sure adjacent ranks are on the same DGX box.” Rank numbering is a fabric-mapping decision; the reshape assumes it.

Two further details reward attention. First, there are two data parallels: the comment distinguishes ExternalDP, “the data parallel group that is not part of the model” where “every dp rank can generate independently,” from the model-side DP where “all the ranks in the same DP group should generate simultaneously … otherwise it will cause deadlock.” That is the difference between the weak-sync replication row and a synchronized axis, expressed as a deadlock condition. Second, decode-context parallelism does not stand alone — when context size exceeds one, DCP “spans PCP first, then TP,” composing with the prefill-context and tensor axes rather than replacing them. Real meshes have axes inside axes, and the file is where the framework admits it.

SGLang’s parallel_state.py draws the same conclusion from the other end: its getters include get_attn_tp_group, get_attn_cp_group, get_moe_dp_group, get_moe_ep_group, and get_moe_tp_group — separate coordinators for attention tensor width and expert placement, because MoE deployments genuinely run different parallel shapes in the attention layers than in the expert layers. The chapter’s “the model has a mesh of dimensions” is not a figure of speech; both engines allocate one communication group per mesh edge. One more construction detail carries an operational lesson: each group initializer asserts its global is unset — assert _TP is None and its siblings — because groups are process-lifetime singletons. Parallelism is decided once at startup and never re-negotiated; changing the mesh means a restart, which is why the decision procedure above earns its paper review — and why a mesh change silently invalidates everything Chapters 8 and 9 cached: tuned kernels and captured graphs are keyed to shapes that no longer exist after the group sizes change.

Map frequent traffic to fast links

A logical mesh becomes a serving topology only when it is mapped to hardware. Keep frequent, latency-sensitive collectives inside the fastest fabric when possible. Put communication that uses larger, less frequent transfers on slower links.

For every parallel dimension, create a small ledger:

Attach message sizes from the target model and batch. The labels alone cannot predict performance — but the ledger plus Appendix A’s transfer formula will get within shouting distance, which is enough to reject most bad meshes on paper. Keep Appendix A’s caveat in view while doing it: concurrent transfers contend for the same fabric, registration and serialization add overhead the formula does not model, and a plan that assumes every collective gets the full link bandwidth will flatter itself.

A decision procedure

Begin by asking whether the weights and required state fit on one device. If they do, test replication first. If weights do not fit, choose between splitting layers and splitting tensors based on links, latency, and available concurrency. If context state does not fit, add a sequence or decode-context dimension. If experts dominate model size, distribute experts and model the all-to-all.

Next, check the workload. Low-concurrency interactive traffic is sensitive to collective latency and pipeline bubbles. High-throughput offline work can fill larger parallel groups. Long-context traffic changes the fraction of memory and communication due to state.

Finally, measure the plan at several batch shapes. Parallel efficiency is not a fixed property of the model. And check one non-performance property before committing: failure scope. A tensor group whose ranks share one switch fails as a unit — Chapter 4’s shared-switch arithmetic means a single fabric fault can zero the capacity of every replica that depends on it. Pipeline stages spread across independent links partition the blast radius but reintroduce the bubble; replication across islands shrinks it furthest at the highest memory price. The parallel plan is also a failure-domain plan, and both reviews deserve to happen on paper, before the first deployment teaches them at production cost.

Quick reference: sizing your first parallel plan

Before working through the full decision procedure, this table gives starting points for common model sizes on a single 8-GPU node with fast intra-node links. Adjust based on your actual link speeds and workload.

These are starting configurations, not answers. Measure at your target batch size and context length before committing. The worked example below shows how to evaluate two legal plans against each other.

Worked example: TP8 or PP2 × TP4

On one fast eight-GPU island, Plan A uses tensor parallel size eight. Plan B uses two pipeline stages, each with tensor parallel size four. TP8 avoids a pipeline bubble but performs layer-frequency collectives across the widest group. PP2 × TP4 confines those collectives to four ranks and sends activations once across the stage boundary; it needs concurrency to keep both stages busy.

For hidden width 8,192 in BF16, the stage-boundary activation is about 16 KiB per sequence token before batching. Walk both plans’ decode traffic at batch 16. Plan B crosses the boundary with 16 sequences × 16 KiB = 256 KiB per step — one transfer per step, latency-dominated like every decode collective, plus the four-rank TP collectives inside each stage. Plan A runs the same 160 per-step collectives but across eight ranks instead of four, paying somewhat longer ring paths on every one; what it never pays is the bubble. So the comparison reduces to a single question: does the concurrency that batch 16 represents keep both Plan B stages busy? At sixteen active sequences the answer is usually yes — each stage always has work — while at batch 2, Plan B idles half its fleet every other step and Plan A’s wider collectives are the smaller evil. Prefill shifts the argument again: chunk activations are megabytes, so the S / b term dominates and Plan A’s eight-rank collectives move real bytes on every layer, while Plan B keeps its collectives on four ranks and crosses the boundary once per chunk. Which plan wins which phase is exactly the kind of question the ledger table exists to structure — and the reason G’s answer lets the phases disagree.

Tensor-parallel message volume depends on the exact sharding and collective algorithm, so derive it from the execution plan rather than copying a generic formula.

Practice: compare two legal meshes

Draw both rank meshes and map them to physical links. For batch 16, calculate the stage-boundary payload and derive the chosen TP collective messages for one prefill and decode step. Predict phase winners at low and high concurrency.

Include memory fit, collective latency, pipeline bubbles, and failure scope. The worked comparison is in Appendix G.

14. Serving Mixture-of-Experts Models

In a dense transformer layer, every token follows the same feed-forward network. In a mixture-of-experts layer, a router chooses a few expert networks for each token. The model gains parameter capacity without applying every parameter to every token: a layer with 64 experts that selects two per token applies roughly 3 percent of its expert parameters to any given token.

For serving, the price of that conditional compute is movement and imbalance. The router’s decision turns one clean matrix multiplication into a scatter: token representations must travel to whichever rank holds the selected expert, and the layer cannot advance until the busiest rank finishes. Two deployments with identical hardware and identical token counts can differ materially in step time on routing alone. This chapter walks the dispatch path, prices the imbalance, and treats expert placement as what it is operationally: a measured feedback loop that moves weights to save per-step time.

Follow one token through an MoE layer

Assume the layer has 64 experts and selects two per token. Experts are spread across eight GPUs. The router produces expert IDs and weights for every token in the batch, so a batch of 8 tokens means 8 hidden vectors in, 16 expert executions, and 16 output vectors combined back into 8.

Each MoE layer dispatches token representations to selected experts. The dispatch boundary is where bytes move; everything left of it is metadata, and everything right of it is per-expert compute whose duration you do not control directly.

flowchart LR
    T["Input tokens"] --> R["Router and top-k selection"]
    R --> D["Dispatch by expert owner"]
    D --> E1["Expert 1"]
    D --> E2["Expert 2"]
    D --> EN["Other experts"]
    E1 --> C["Combine weighted outputs"]
    E2 --> C
    EN --> C
    C --> O["Layer output"]

The runtime then performs four steps:

1. group or pack token representations by selected expert. This is a permutation, usually built from a histogram or sort over the expert IDs: kernels that follow want contiguous per-expert segments, not scattered rows;
2. send each representation to the rank that owns the expert;
3. execute the expert networks, typically as a grouped matrix multiplication in which one kernel processes every expert’s segment as a batched matrix with its own row count;
4. return and combine the expert outputs using the router weights — the inverse permutation followed by a weighted sum.

The two network phases are usually called dispatch and combine. Across an expert-parallel group they resemble all-to-all communication, although specialized implementations may use custom point-to-point patterns.

Price one dispatch

Take hidden width 8,192 in BF16, so each routed activation is 2 bytes × 8,192 = 16 KiB. In a decode step with 64 active sequences and top-2 routing, the batch creates 128 assignments, and dispatch moves 128 × 16 KiB = 2 MiB per layer — paid again by combine, so 4 MiB per layer per step. A dense layer of the same width moves none of this: its 64 rows stay local and read shared weights.

Prefill multiplies the payload by sequence length. A 4,000-token prompt creates 8,000 assignments, so dispatch alone carries 8,000 × 16 KiB = 125 MiB per layer in one direction. Multiply by the number of MoE layers to see why the interconnect, not the expert compute, can dominate prefill step time — and why expert-parallel prefill wants the highest-bandwidth tier of the fabric available.

The router’s own output is tiny by comparison — two expert IDs and two weights per token — but it decides the permutation, and the permutation decides the message boundaries. Observability should follow the same rule the payload does: count assignments, not unique input tokens. A trace that records 64 tokens for the batch above underreports the movement by half.

The slowest expert sets the pace

Routing is not perfectly balanced. A programming workload may favor different experts from a multilingual chat workload. Even within one batch, a few experts can receive many more tokens than others.

All ranks must finish before the layer can advance. A GPU that owns a hot expert becomes the straggler while other GPUs wait. Average balance over an hour does not prevent step-level imbalance: a workload can be perfectly balanced in aggregate and still send 3× the mean to one expert every step. For streaming requests the straggler is invisible as an average but visible as latency: every active sequence’s next token waits for the busiest rank, so step-time variance from routing shows up directly as inter-token latency variance.

Padding each expert to a fixed capacity creates regular shapes but wastes work. Dropping excess tokens can change model quality. Dynamic grouped GEMM avoids some padding but must handle many small or uneven matrices.

vLLM’s EPLB logging reduces rank balance to one number per step: balancedness = avg_tokens / max_tokens, the ratio of the mean per-rank load to the busiest rank’s load, summed across layers. A perfectly balanced deployment logs 1.0; the worked example at the end of this chapter logs roughly 0.44. The number is cheap to compute and belongs in MoE dashboards next to per-expert histograms.

Capacity, padding, and the drop decision

Use the worked example’s counts — 22, 14, 7, 6, 5, 4, 3, 3 across eight experts, 64 assignments, a mean of 8 per expert. A common mitigation is a fixed per-expert capacity, sized as a multiple of the mean. At 1.25× mean, each expert gets 10 slots: 80 slots for 64 assignments, and expert 0, which brought 22, overflows by 12. Those 12 tokens are either dropped — their outputs for this layer become zeros or pass-throughs, which is a model-quality decision — or the layer must handle a ragged shape anyway.

Padding to the maximum instead of dropping means sizing every expert for the busiest one: 22 slots × 8 experts = 176 assignment slots for 64 useful ones. About two thirds of the expert compute is padding. That is the steady-state tax of an imbalanced placement, paid every step.

Dynamic grouped GEMM removes the padding by executing exactly the segments that exist, at the cost of ragged shapes that vary per step — which is precisely what complicates the CUDA-graph capture decisions of Chapter 9. Expert-parallel load balancing attacks the problem one level up: change the placement so the maximum falls, and neither drop nor pad.

Expert placement is a cache problem

The simplest placement gives each expert one owner. Popular experts overload their ranks. Replicating selected experts trades additional weight memory for more destinations and better balance.

Expert load balancing is a measured placement feedback loop. The loop moves weights between steps; a placement only pays for itself if the straggler time it removes exceeds the weight movement and cache disturbance it causes.

flowchart LR
    X["Router trace"] --> L["Tokens per expert and rank"]
    L --> P["Candidate placement"]
    P --> M["Weight movement under new generation"]
    M --> V["Validate straggler and goodput change"]
    V --> X

A placement controller can collect routing statistics and periodically move or replicate experts. This is expert-parallel load balancing, often abbreviated EPLB. Changes must be coordinated: routers need the new location map, weights must be available before traffic moves, and in-flight batches must finish under a consistent mapping.

Reacting too quickly to a noisy batch can create movement churn. Reacting too slowly leaves hotspots. Use a stable observation window and include the cost of reconfiguration.

At the pinned revisions, vLLM implements EPLB state, policy, communication, and rebalance execution under vllm/distributed/eplb. SGLang’s corresponding manager, algorithms, distribution tracking, and location updates live under sglang/srt/eplb.

From counts to a placement

vLLM’s rearrangement policy is vllm/distributed/eplb/policy/default.py, adapted from DeepSeek’s EPLB. Its module docstring in eplb_state.py fixes a four-term vocabulary worth adopting verbatim: a logical expert is part of the model; a redundant expert is an extra copy created for balancing; a physical expert is any replica instantiated on a device; a local physical expert is one on the current device. The docstring’s example: DeepSeek-R1 has 256 logical experts, adding 32 redundant experts gives 288 physical experts, and 32 EP ranks hold 288 / 32 = 9 local physical experts each.

The input is a load tensor of shape [layers, num_logical_experts]. The entry point rebalance_experts first aggregates: in EplbState.rearrange, per-rank physical loads are mapped back to logical experts with a scatter_add_ over physical_to_logical_map, summed over the observation window, and all-reduced across ranks. Then the policy runs on the host — the comment is explicit that “the load window and current map have to come back” to CPU.

The policy itself is two greedy passes plus a topology split:

• balanced_packing sorts experts by load descending and repeatedly assigns each to “the lightest pack; full packs are masked out by inf” — longest processing time first, the classic list-scheduling heuristic.
• replicate_experts grows logical experts into physical slots one at a time, each round replicating argmax(weight / logcnt) — the expert whose load per existing replica is highest. Replication stops when physical slots run out.
• rebalance_experts_hierarchical runs three steps when the expert groups divide evenly across nodes: pack groups to nodes, replicate within nodes, then pack physical experts to GPUs, dividing each logical load by its replica count first (“Effective per-physical load = logical load divided by replica count”). When divisibility fails, the caller degenerates to global balancing by invoking the same function with one group and one node.

Two details matter for operations. First, preserve_intragpu_slots post-processes the new mapping “so that experts that remain on the same GPU keep their previous slot positions when possible” — an expert that did not change ranks is not copied at all. Second, EplbState.step does not record the load window continuously: _should_record_current_step enables recording only when the next rearrangement (or the next logging step) is within expert_load_window_size steps. The window is a ring buffer that is allowed to hold stale entries most of the time, because only its freshest window_size entries are ever read. And dummy steps still advance the rearrangement counter — the comment explains why: “to ensure all ranks are performing collective communication.” A rank that skips a collective while others take it deadlocks.

Installing a placement while serving continues

A new mapping is only useful once its weights are in place, and the weight movement must not stop serving. The two pinned implementations solve this differently, and both are worth reading.

SGLang’s eplb_manager.py makes rebalancing a coroutine. on_forward_pass_end calls next() on a generator whose loop is, in effect: serve for eplb_rebalance_num_iterations forward passes, then run one rebalance. The rebalance itself yields too — layers are updated in chunks of eplb_rebalance_layers_per_chunk, and rebalance yields between chunks, so each engine step installs one more chunk and serving continues between installs. The mapping update in expert_location.py is correspondingly per-layer: ExpertLocationMetadata.update builds a layer mask and applies torch.where(mask_update, other_field, self_field), so layers outside the current chunk keep the old mapping. Both directions of the map — physical-to-logical and logical-to-all-physical, plus the CPU copies — flip together for the layers in the chunk. A batch never sees a layer whose two maps disagree.

The manager also refuses to rebalance blindly: _check_rebalance_needed skips the update when the windowed average GPU utilization exceeds eplb_min_rebalancing_utilization_threshold, and the constructor asserts that the rebalance interval is at least the distribution recorder’s circular buffer size — “Otherwise, the circular buffer will contain stale data.” When a chunk’s peer-to-peer transfer cannot find a rank holding a needed replica (p2p_missing_logical_experts is non-empty), update_expert_location_with_recovery falls back to a DRAM backup client or a disk reload filtered through generate_weight_name_filter to just the missing experts. After an elastic scale-up, the pattern changes: only rank 0 computes the new mapping — the comment explains that one owner keeps “process-local launch topology” out of the decision — and the result is broadcast to the expanded world, so every rank installs an identical map.

vLLM moves weights through pre-allocated buffers instead of yielding. rebalance_execute.py defines move_to_buffer and move_from_buffer around transfer_layer, which works one MoE layer at a time on weight tensors of shape (num_local_physical_experts, hidden_size_i) — for a linear expert, typically two tensors, up and down projection. Much of the time nothing needs to move: transfer_layer returns a TransferMetadata carrying is_unchanged and is_received_locally masks, and experts whose physical slot did not change are skipped — the same no-op-copy goal as vLLM’s preserve_intragpu_slots, applied at transfer time. In async mode the transfer runs on a side CUDA stream into expert_weights_buffer, and the main loop only commits when EplbState.step sees rebalanced and _all_ranks_result_ready both true, at which point _move_to_workspace swaps the staged weights in. If the next rearrangement boundary arrives while an async rearrangement is still in flight, step returns early without resetting the counter — the rearrangement is deferred, not skipped. The profile path is also instructive: is_profile performs a “dummy rearrangement with maximum communication cost” so that profile_run reserves memory for the communication buffers before real traffic arrives.

Prefill and decode need different communication

Prefill sends many tokens through an MoE layer. Large messages can use network bandwidth efficiently, and throughput-oriented dispatch kernels are appropriate.

Decode may send only one token per active sequence. Messages are smaller and latency dominates. A communication method tuned for large prefill transfers may perform poorly here. The 2 MiB decode dispatch from the pricing section is 128 small contributions that must be assembled per rank; the 125 MiB prefill dispatch is a bulk transfer where startup latency is noise.

Appendix A’s transfer model shows how different the two regimes are on the same fabric. With the declared NVLink-class figures (a = 20 µs, b = 450 GB/s), the 2 MiB decode dispatch takes 20 µs + 2.1 MB / 450 GB/s ≈ 25 µs — four fifths of it is the startup term. The 125 MiB prefill dispatch takes 20 µs + 131 MB / 450 GB/s ≈ 310 µs — nine tenths of it is bandwidth. Tuning that helps one regime barely touches the other, which is why the low-latency and high-throughput paths exist as separate code paths rather than one kernel with a flag.

The official DeepEP repository makes this distinction explicit through expert-parallel dispatch and combine primitives designed for high-throughput and low-latency regimes. Current vLLM deployment documentation likewise describes separate communication choices for prefill and decode in its expert-parallel guide.

The placement layer has not caught up with the communication layer. vLLM’s rearrange carries a standing TODO(bowen): Treat differently for prefill and decode nodes — at the pinned revision, one placement serves both phases even though their traffic patterns differ. Until that changes, phase-aware treatment lives in the communication choice, not the placement.

This is another reason phase disaggregation can help: each pool can choose the parallel and communication plan suited to its phase.

Overlap communication with useful compute

MoE execution contains work that can sometimes overlap. Token dispatch for one batch can run while another batch computes experts. Shared experts — weights applied to every token, with no router and therefore no dispatch — can execute on a different stream from routed experts while the routed tokens are still in flight. Combine for an earlier layer or batch can overlap later work when dependencies allow it.

Systems use names such as two-batch overlap, dual-batch overlap, or single-batch overlap for different schedules. The name matters less than the timeline.

Draw the operations and dependencies:

batch A: route -> dispatch -> expert compute -> combine
batch B:          route -> dispatch -> expert compute -> combine

Overlap improves utilization only if the tasks use compatible resources. A communication kernel that consumes many streaming multiprocessors can compete with expert GEMM. Extra in-flight batches also need more buffers and complicate cancellation and failure: a batch that is aborted mid-dispatch leaves partially filled receive buffers that the schedule must reclaim.

The network topology is visible in every MoE layer

If expert ranks span nodes, each MoE layer sends token activations across the network. Group-limited routing can encourage local destinations, but changes the model’s routing behavior and must be part of the architecture.

Replication gives the dispatcher a choice, and SGLang turns that choice into a precomputed table. compute_logical_to_rank_dispatch_physical_map in expert_location.py builds, for every rank, layer, and logical expert, which physical replica that rank should prefer. _find_nearest_expert encodes the preference order: if there is only one candidate, take it; otherwise prefer a replica on the same GPU, then one on the same node — but only “when it narrows the candidate set”; otherwise return −1. Ranks left with −1 are assigned by _fair_choices, a seeded shuffle that spreads their traffic evenly over the replicas, and an assertion verifies no −1 survived. The table is computed once per placement generation, so topology preference costs a lookup per assignment rather than a decision per token.

Place experts and ranks with network rails in mind. A hot expert behind one NIC can bottleneck several GPUs. Measure dispatch and combine by source, destination, message size, and layer. Aggregate network bandwidth can look healthy while one rail determines step time.

Worked example: balance the busiest rank

An eight-expert layer routes 64 tokens with counts 22, 14, 7, 6, 5, 4, 3, 3. With two contiguous experts per rank, rank 0 receives 36 assignments while rank 3 receives six. The four rank loads are 36, 13, 9, 6; the mean is 16, so vLLM’s balancedness statistic would log 16 / 36 ≈ 0.44 — the layer finishes at less than half its average utilization.

Repack the pairs: (E0, E7) = 25, (E2, E3) = 13, (E4, E5) = 9, (E1, E6) = 17. The maximum falls from 36 to 25, about a third less work on the hot rank, while average utilization barely changes — it cannot, the total is fixed. Only the maximum gates the layer.

What the repack costs: experts E1 and E7 swap ranks, so two experts’ weights cross the fabric. Assume each expert’s weights total 1 GiB. Using the declared NVLink-class figures from Appendix A, transfer time = a + S/b with a = 20 µs and b = 450 GB/s gives 20 µs + 1.07 GB / 450 GB/s ≈ 2.4 ms per expert, about 5 ms once for the pair. If the lighter hot rank saves even 0.25 ms per step, the movement pays for itself in 20 steps; if the workload drifts and the saving is 0.05 ms per step, payback takes 100 steps and the next drift may reverse the decision first. That sensitivity is why the observation window and the rebalance interval are configuration, not afterthoughts.

Replication is the alternative worth pricing against the repack: keep E0’s contiguous pair and copy E0 onto rank 3, splitting its 22 assignments between the two replicas. At an even split, rank 0 falls to 25 and rank 3 rises to 17 — the same maximum as the repack, from one expert copy (≈2.4 ms) instead of two. The trade is memory: the replica holds 1 GiB for as long as the placement lives, and every placement generation that keeps E0 hot re-decides whether that GiB is still earning its keep.

At hidden width 8,192 and BF16, each routed activation is 16 KiB. Top-2 routing creates 128 assignments for 64 input tokens—roughly 2 MiB before protocol overhead in each dispatch/combine direction. A placement that balances compute but increases slow-link traffic can still lose: moving an expert from a co-located rank to a cross-node replica converts local hops into network hops for every assignment it serves.

Practice: replay and update a placement

Replay the counts above on four ranks with two experts each. Propose a new placement, calculate per-rank assignments and activation payload, and predict the straggler. Compute the balancedness ratio for both placements and check it moves the way the maximum moves. Then design a generation-safe EPLB update that cannot mix old and new mappings within a batch, and decide which layers you would move first if the update had to be chunked.

Compare prefill and decode traces and include weight-movement cost. The worked placement is in Appendix G.

Expert serving makes phase differences especially pronounced. Chapter 15 generalizes the idea of assigning different stages to different worker pools.

15. Stage Disaggregation: Encoder, Prefill, and Decode

A colocated LLM worker performs both prefill and decode. That arrangement keeps state local and makes one worker responsible for the whole request. It also forces two different workloads to share the same queue, hardware, and parallel configuration — and the two workloads disagree on all three. Prefill wants large batches and compute efficiency; decode wants steady, small steps and memory bandwidth. Sharing a queue means one of them is always mis-served.

Disaggregated serving separates stages so each can be scheduled and scaled independently. The common case places prefill on one worker pool and decode on another. The price is a new stage in the middle — KV state must move between pools — and a new coupling: neither pool can be sized correctly without measuring the other.

Why split prefill from decode?

Prefill benefits from large compute-efficient operations. Decode values steady, low-latency steps and sufficient memory bandwidth. A long prompt running beside active decoders can create an output stall. A tensor-parallel plan that helps prefill may add too much synchronization to decode.

Prefill/decode separation turns one engine queue into a stage pipeline. The feedback edge on the right is what keeps the pipeline honest: admission must know whether decode capacity actually exists, not whether prefill is idle.

flowchart LR
    A["Admission"] --> P["Prefill queue and workers"]
    P --> K["KV state transfer"]
    K --> D["Decode queue and workers"]
    D --> O["Output stream"]
    D --> B["Decode capacity signal"]
    B --> A

The stall is not a corner case — it falls out of the service-time model directly. Using the worked example’s prefill_ms = 20 + 0.035 × tokens, one 6,000-token prompt occupies its worker for roughly 230 ms. Every decode step that would have run during those 230 ms is late, so every sequence sharing that worker sees an inter-token gap larger than the entire prefill. Against an ITL budget of 150 ms, a single long prompt on a colocated worker is by itself an SLO breach for all its neighbors. Chunked scheduling softens the spike by interleaving prefill and decode steps, but the bytes still compete for the same compute and the same memory bandwidth.

With separate pools, a prefill worker processes the prompt and produces the initial KV state. That state moves to a decode worker, which generates the rest of the response.

request -> prefill queue -> prefill workers
                              |
                           KV transfer
                              |
                              v
          stream <- decode workers <- decode queue

Separation also makes each phase’s parallel plan a free choice. Prefill can use wider tensor parallelism to shorten the 230 ms; decode can stay at a narrow plan where synchronization overhead dominates. Chapter 13 priced the difference: two collectives per layer means 160 collective launches per step at any tensor width above one, costing ~3.2 ms of startup alone when decode batches are small — overhead decode pays every 45 ms step, while prefill’s large payloads amortize it. Colocated deployment forces one compromise on both phases.

The DistServe and Splitwise papers study phase splitting as a way to reduce interference and select phase-specific resource plans.

The transfer is part of latency

The KV cache for a long prompt can be large. Disaggregation helps only if the state can be transferred, registered, and made visible before the saved interference or better placement pays for that cost.

One transfer, four commitments. The destination reserves before data moves, the source publishes into the reservation, both sides poll for completion, and only a verified transfer inserts the request into a decode batch. Every earlier state is reversible; insertion is not.

flowchart LR
    R["Request admitted to decode"] --> A["Reserve destination blocks"]
    A --> PB["Publish block map to prefill"]
    PB --> T["Prefill transfers into reserved blocks"]
    T --> V{"Poll: Success on all ranks?"}
    V -->|Yes| I["Insert into decode batch"]
    V -->|No or Failed| X["Release reservation; retry or re-prefill"]

A transfer protocol needs several pieces of metadata: request identity, model and cache format, source and destination ranks, block ranges, memory addresses or handles, and completion state. The sender and receiver must agree on how tensor-parallel or context-parallel shards map between them. Real deployments add wrinkles the metadata must survive: prefill and decode pools running different attention-TP sizes, pipeline stages that own only a layer range, and cache layouts that are not flat layer-indexed lists.

Bulk data should move on a path designed for it, while control messages arrange the rendezvous. A push design lets the prefill side initiate. A pull design lets decode request the blocks. Both need timeouts, cancellation, and idempotent cleanup.

If transfer fails, the system can retry, choose another decode worker, recompute prefill, or fail the request. The policy should depend on the remaining deadline and expected recompute cost. The costs are computable. Retrying the transfer costs another ~95 ms of link time plus whatever queue delay applies; recomputing the prefill costs the full ~230 ms again plus the freed worker’s opportunity cost; failing the request spends everything already paid. Against the 600 ms TTFT budget, a request that has already waited 200 ms in queues cannot afford recomputation and barely affords one retry — which is why the policy belongs to admission, which knows the elapsed budget, not to the transfer layer, which does not.

Chunked sends hide the boundary

Nothing requires the full 1.83 GiB to exist before transfer starts. SGLang’s sender interface exposes the seam directly: should_send_kv_chunk(num_pages, last_chunk) decides per step whether ready pages go out now, and the prefill scheduler’s send_kv_chunk path streams completed blocks while later chunks are still computing. The default is eager — “return num_pages > 0” — and each chunk’s readiness is tracked separately until a final last_chunk=True send concludes the request.

The overlap is worth real milliseconds. Suppose chunks complete every 2,000 tokens. The first chunk is ready at 20 + 0.035 × 2,000 = 90 ms and its ~610 MiB take 12 + 610 MiB / 22 GiB/s ≈ 39 ms on the link — finished at ~129 ms, while prefill still has 140 ms of compute left. The last chunk leaves at ~230 ms and needs only its own slice of link time, so the pipeline’s transfer tail shrinks from the serial 95 ms to roughly 12 ms plus the final chunk’s bandwidth time, about 40 ms total. End-to-end, chunked transfer lands near 270 ms instead of 325 ms — without changing a single byte moved. The trade is protocol state: partial transfers hold reservations longer, and every retirement path must clean up pending chunk bookkeeping or leak it.

Five states from handshake to insert

SGLang’s disaggregation stack is built from five small abstract roles in base/conn.py: a BaseKVManager holding transfer state, a BaseKVSender, a BaseKVReceiver, and a BaseKVBootstrapServer that lets pools find each other. The transfer’s entire lifecycle is one poll enum:

KVPoll: Failed=0, Bootstrapping=1, WaitingForInput=2, Transferring=3, Success=4

The scheduler polls senders and receivers every step; nothing blocks. The KVArgs the manager carries are deliberately raw — kv_data_ptrs, kv_data_lens, kv_item_lens, layer ids, an ib_device — because the transports move GPU memory by address, not by copying through intermediate buffers. The comments catalog the layout hazards metadata must express: per-tensor TP slice dims “used when prefill/decode attn_tp_size differ”, prefill_start_layer/prefill_end_layer for pipeline-parallel sub-ranges, and auxiliary state types (MAMBA, SWA, swa_ring) beyond plain KV.

Two details reward attention. First, the handshake is reserve-then-publish: the decode side’s DecodePreallocQueue allocates destination blocks before data moves and tracks “_num_published_destinations — destinations visible to prefill but not yet on the transfer queue.” Prefill never sends into blocks decode has not committed. Admission into the handshake is itself queued: requests wait in a PrefillBootstrapQueue until the bootstrap server resolves pool endpoints, and _check_if_req_exceed_kv_capacity rejects up front any request whose KV indices cannot fit the destination pool — failing fast instead of mid-transfer. Second, completion is a distributed agreement, not a local fact: process_disagg_prefill_inflight_queue all-reduces the poll states across the attention TP/CP groups, and in pipeline parallelism a later rank treats a non-terminal poll as “undone” when an earlier rank already saw Success/Failed — the comment attributes the mismatch to “clock skew or propagation delay” and chooses to wait rather than crash.

The failure path is equally explicit. A Failed poll routes to handle_inflight_transfer_failure, which releases the request’s KV reservation, aborts the request with an internal-server-error status, increments a transfer_failed_reqs counter, and streams the error to the client. A separate optimistic path — optimistic_release_and_requeue, gated by should_force_retry — releases the destination and re-queues the request for another prefill attempt instead of failing it. And one bookkeeping rule prevents a slow leak: clear_pending_chunk_send must run on every path that retires a request without a final chunk, because “a stale entry holds the unified-memory compaction gate closed for the process lifetime.”

Observability is part of the interface: get_transfer_metric returns KVTransferMetric with transfer_latency_s, alloc_latency_s, and transfer_total_bytes — and the docstring admits “backends that cannot isolate transfer latency can leave this as None.” A dashboard that averages over Nones silently lies; the schema makes the gap visible instead. The poll loop itself is honest about its rough edges — one inline comment reads “todo: set Transferring correctly in backend,” so callers treat WaitingForInput and Transferring alike as “still in flight.” And on the decode side, insertion is FIFO through pop_preallocated, which refuses to run at all under pipeline parallelism unless the caller supplies consensus rids — the error message says it plainly: “PP consensus is required when pp_size > 1.”

Two queues create a coupled system

Separating stages does not remove queueing. It creates a queue before prefill, a transfer boundary, and a queue before decode.

If prefill produces requests faster than decode can consume them, completed KV state accumulates while users wait for a decode slot. Scaling prefill harder would make the system worse. If decode is overprovisioned, expensive workers wait for prefills.

The correct pool ratio depends on arrival rate, prompt and output lengths, cache hits, and each phase’s service time. Measure the whole pipeline. A low prefill queue can hide a growing decode queue, and the symptoms are specific: completed-but-uninserted KV state holding reservations, rising transfer-queue age, decode admission lagging prefill completion. Each is the signature of prefill outproducing decode admission, visible one stage before users feel it.

Raw throughput may fall after adding transfer while goodput rises because TTFT and ITL become more predictable. State the metric. “Disaggregation increases throughput” and “disaggregation never increases throughput” are both too broad.

Sizing the pools

Appendix A’s Q = λW makes the coupling concrete. Assume the worked example’s 6,000-token prompts, decode steps of 45 ms, batches of 32, and 400-token outputs. One prefill worker completes a request every ~230 ms, so its production rate is λ_p ≈ 4.3 requests per second per worker. A decode worker retires its whole batch only when sequences finish: with 400-step generations, each sequence occupies a slot for 400 × 45 ms = 18 s, so a 32-slot decode worker turns over 32 / 18 s ≈ 1.8 requests per second. Steady state needs prefill production to match decode turnover: one prefill worker feeds roughly two decode workers at this mix. Every parameter moves the ratio — longer outputs raise decode residency, cache hits raise effective prefill capacity, and a burst of short prompts can flip the bottleneck within minutes.

The transfer stage sits between them with its own queue and its own capacity: at 12 ms setup plus 22 GiB/s, concurrent transfers share a pipe that a 6,000-token prompt occupies for ~95 ms. Ten concurrent such transfers serialize into a second of link time. Admission should treat transfer bandwidth the way Chapter 5 treats GPU occupancy — a schedulable resource with its own queue, not a free side effect of finishing prefill.

Conditional and dynamic disaggregation

Not every request should cross a stage boundary. A short prompt may be cheaper to run entirely on a colocated worker. A request whose prefix is already cached on a decode worker may skip remote prefill. A large prompt with a tight ITL SLO may benefit most from separation.

Conditional placement chooses between local reuse and a transfer boundary.

flowchart TB
    R["Request shape and queue state"] --> X{"Split saves more interference than transfer costs?"}
    X -->|No| C["Colocated prefill and decode"]
    X -->|Yes| P["Remote prefill"]
    P --> T["Versioned KV transfer"]
    T --> D["Reserved decode slot"]

A conditional router compares local execution with remote prefill plus transfer and queueing. The worked example gives the comparison shape: remote costs ~230 ms prefill + ~95 ms transfer + a decode-queue wait; colocated costs ~230 ms prefill on a worker whose decode neighbors each absorb an SLO-breaching stall. The transfer is worth it when the stall it removes — spread across the sequences that would have shared the worker — exceeds the 95 ms and the added queueing. Short prompts flip the comparison: at 500 tokens, prefill is ~38 ms and the KV state only ~156 MiB, but the transfer setup alone is 12 ms, nearly a third of the phase it enables.

A dynamic system can also change pool membership as workload phase ratios change. The comparison the router runs, per request class:

The last row is the honest one: under low load, disaggregation adds latency and removes nothing, because there is no interference to remove. Conditional placement is not a per-request optimization only — it is how the deployment stays correct across load levels. Reconfiguration must account for warm weights, graph capture, cache loss, and draining — the same invalidation cascade as Chapter 13’s mesh changes, because a worker changing pools changes its parallel plan.

Encoder, prefill, and decode

Multimodal models add an encoder stage. Large video or vision encoders can dominate first-output latency and use different hardware shapes from language decode. Separating encoder, prefill, and decode produces an E/P/D topology.

Encoder outputs must now move to prefill, followed by KV state moving to decode. The two movements differ in kind: encoder output is a fixed-size embedding set per image — the same size whether the caption is five words or five hundred — while prefill KV scales linearly with text. Different elasticity is exactly why one shared pool serves both poorly: the encoder pool sizes for media throughput, the prefill pool for token throughput, and neither’s queue says anything useful about the other. The additional boundary is worthwhile only when independent batching, caching, or hardware assignment pays for it. Repeated questions about the same media can make an encoder cache especially valuable: the encoder output for an unchanged image is deterministic, so a hit removes the dominant first-stage latency without touching the language pipeline.

The idea generalizes beyond language. A diffusion pipeline can place its text encoder, denoiser, and decoder in different pools. A rollout system can separate generation from training and weight distribution. Disaggregation is stage placement, not a feature unique to KV caches.

What the implementations reveal

vLLM defines connector interfaces and several transfer or offload integrations under distributed/kv_transfer. SGLang’s prefill, decode, staging, and transport implementations live under srt/disaggregation. The current deployment surfaces are documented in the official vLLM disaggregated-prefill guide and SGLang PD guide.

Both code trees contain several backends and compatibility checks. That is a useful warning: a transport name alone does not guarantee support for a model’s cache layout, parallel sizes, speculative mode, or device. vLLM’s connector factory registers the integrations by name, and the names themselves teach the design space: NixlConnector alongside separate NixlPushConnector and NixlPullConnector variants — this chapter’s push-versus-pull choice made literal — plus MultiConnector for composing several tiers and OffloadingConnector for moving KV to host memory rather than another GPU pool.

The Mooncake paper extends the design around a KV-centric architecture using GPU, CPU, memory, and storage resources. It also highlights early rejection under overload, connecting disaggregation back to goodput and admission control.

One interface, two sides

vLLM’s kv_connector/v1/base.py organizes the same lifecycle as one class with two halves, and its module docstring is the cleanest statement of the boundary. Scheduler-side primitives decide what should happen: get_num_new_matched_tokens reports how many tokens exist in a remote cache — with the explicit contract that it “might be called multiple times for a given request and should be side-effect free,” because the scheduler probes speculatively; update_state_after_alloc reacts to buffer allocation; request_finished decides who owns the blocks now — it “returns whether KV cache should be freed now or if the connector now assumes responsibility for freeing the blocks asynchronously,” which is how a cache tier can outlive its request; take_events exports the same KV events Chapter 16’s distributed cache consumes.

Worker-side primitives do the moving, at layer granularity: start_load_kv begins async loads, wait_for_layer_load blocks until layer i has arrived — so transfer overlaps the forward pass instead of preceding it — and save_kv_layer/wait_for_save mirror the pattern on the way out. handle_preemptions lets the connector react when the scheduler retracts requests whose blocks are mid-transfer, and get_finished reports which async sends and receives have completed so the scheduler can act on them. Capability is declared, not discovered: requires_kv_delivery states “whether this connector hands off KV that must be reliably delivered,” and a SupportsHMA marker class flags connectors that handle hybrid memory attention — the machine-checkable form of the compatibility warning above. The split enforces a rule this chapter has been circling: scheduling decisions live in one process, bytes move in another, and metadata is the only thing that crosses.

Worked example: price the state boundary

Use prefill_ms = 20 + 0.035 × tokens, a 45 ms decode step, and a KV link with 12 ms setup plus payload at 22 GiB/s. The 6,000-token Atlas prompt creates about 1.83 GiB of KV state — 6,000 × 320 KiB = 1,875,000 KiB. Ideal transfer is therefore roughly 95 ms, compared with about 230 ms of prefill.

Walk the placement decision for one such request. Remote: 230 ms prefill (assuming a free prefill worker) + 95 ms transfer + one decode step of 45 ms before the first token — about 370 ms of pipeline time before output starts, against a 600 ms TTFT budget that leaves roughly 230 ms for both queues. Chunked sends at 2,000-token granularity pull that to roughly 315 ms — the transfer overlaps prefill instead of following it — and hand back 55 ms of queue budget without moving one byte less. Colocated on an otherwise idle worker: 230 ms and no transfer — but “idle” is the assumption that fails under load. The same request placed beside eight decoding sequences inflicts a 230 ms stall on all of them; eight sequences × one breached 150 ms ITL budget is the interference cost the transfer is buying down. Disaggregation wins here not because 95 ms is cheap but because the colocated alternative is worse for everyone sharing the worker.

Now give the transfer a failure. If the link drops at the moment prefill finishes, the request has ~370 ms of pipeline time committed and roughly 230 ms of TTFT budget left. A retry costs another 95 ms plus queueing — feasible if the decode reservation survived, impossible if it was released. Recomputation costs 230 ms of prefill again plus re-transfer — over budget. The right policy at this point is a bounded retry with reservation held; the wrong time to decide that is during the failure. Deadlines belong in the admission record so the failure handler can compare costs instead of guessing.

That boundary is material. Disaggregation wins only if isolating decode from prefill interference and improving pool utilization repays the transfer and a new queue. An idle prefill worker is not capacity when no decode slot will be available afterward.

Practice: simulate three placements

Build prefill, transfer, and decode service-time tables from the functions above. Compare colocated, always-disaggregated, and conditional placement for short and 6,000-token prompts under bursts.

Report every stage queue, TTFT, ITL, goodput, transferred bytes, failure cleanup, and idle capacity. Derive a conditional split threshold. The worked model is in Appendix G.

16. Hierarchical and Distributed Model-State Caching

One worker finishes processing a 40,000-token document. Ten minutes later, another request asks a new question about the same document—but the router sends it to a different worker. The first worker has the useful KV state. The second has free capacity. The first worker’s cache cannot help the second, and the second’s idle compute cannot help the request except by redoing work that was already done correctly somewhere else.

A local prefix cache cannot satisfy both goals. A distributed cache makes state visible beyond one GPU, but turns reuse into a placement, transfer, and consistency problem. The cache stops being a data structure and becomes a small distributed system with its own failure modes — which is why this chapter spends more time on publication protocols and invalidation than on hit rates.

Recompute, retain, or transfer

Every reusable state object presents three choices. The service can discard and recompute it, keep it near the producer, or move it to a tier where another consumer can retrieve it.

The decision depends on four quantities:

expected reuse value
- storage cost
- transfer cost
- management and failure cost

The three costs are computable for real state, and they are not close. Recompute price comes straight from Appendix G’s standing constant: at 0.06 ms per token, recomputing a 40,000-token document costs 2.4 seconds of GPU time. Transfer price uses Chapter 15’s KV link: the document’s 40,000 × 320 KiB = 12.2 GiB move in about 565 ms at 22 GiB/s including the 12 ms setup. Retention price is the tier’s capacity times how long the state waits — 12.2 GiB parked in host memory is memory that cannot back active sequences. So for long documents the ordering is stable across any plausible reuse probability: transfer beats recompute by 4×, and both beat losing the state. For short prefixes the ordering flips — a 500-token prefix costs 30 ms to recompute, and moving its 156 MiB buys nothing unless another request arrives before it would have been evicted anyway.

Popularity and reuse delay matter because a valuable object held too far in advance displaces other state. Expected value is reuse probability times recompute savings, discounted by how late the reuse arrives — the same shape as any caching economics, applied to tensors instead of pages.

The three prices side by side, using the constants above:

The middle row is Chapter 15’s boundary seen from the cache’s side of the glass, and the setup term matters more than the byte count for small state: a 156 MiB transfer spends 12 ms of its 19 ms on setup, so tiny prefixes travel badly no matter how fast the link is.

A cache can have several tiers

A practical hierarchy might include GPU memory, host memory, local NVMe, and a remote memory or storage service. The fastest tier holds active request state and the hottest reusable blocks. Lower tiers trade access time for capacity.

A hierarchical cache trades increasing capacity for increasing access cost.

flowchart LR
    G["GPU KV blocks"] <--> H["Host memory"]
    H <--> L["Local storage"]
    L <--> R["Remote cache"]
    R <--> O["Durable object storage"]

Promotion moves a block toward the GPU when reuse becomes likely. Demotion or write-back preserves an evicted block in a lower tier. Prefetch begins a load before the request reaches execution. A write-through policy backs up state as it is created; write-back waits until eviction or another trigger.

Each policy moves bytes at a different time. Write-through adds traffic to the foreground path but leaves a ready copy. Write-back avoids copying cold state and may delay eviction. Prefetch hides latency when prediction is correct and wastes bandwidth when it is not.

The rows expose the real constraint: all three policies spend the same bandwidth pool, so the choice is not whether to pay but when — and “when” determines which failure mode you must engineer cleanup for.

At the pinned SGLang revision, hiradix_cache.py implements host and storage coordination, prefetch, backup, write policies, and eviction around the radix index. vLLM exposes a connector and offloading model with scheduling, events, metrics, and workers under its offloading package.

The implementations differ, but both show that an external cache is an asynchronous subsystem rather than a larger dictionary.

Two tier implementations

SGLang’s HiRadixCache extends the ordinary radix cache with a second pool: at construction it builds token_to_kv_pool_host, sized from the server’s hicache_ratio and hicache_size settings relative to the GPU pool. The constructor is also a reminder that cache tiers inherit every layout question: each attention-pool type gets its own host-pool class — MHA, MLA with its own DCP rank parameters, DSA and sparse variants filled in by later attachment — and anything else fails fast with “HiRadixCache only supports MHA, MLA, DSA, and MSA models.” Every demotion and promotion then becomes a copy between two allocators the cache owns, tracked per node. Which direction moves bytes when decides the policy name: in write_through mode, write_backup copies a finished node to host as part of normal operation and eviction just drops the GPU copy of anything already marked backuped; the _evict_write_through docstring states the invariant plainly — “drop non-backuped leaves, demote already-backuped ones. Nothing is staged to host during eviction.” In write_back mode eviction itself does the staging, and the code carries a warning worth heeding: “note this path will be deprecated in the future.”

Eviction walks a heap of evictable_leaves ordered by the strategy’s priority, skips any node with lock_ref > 0 — in-use prefixes simply do not evict — and calls _promote_parent afterward: once all of a node’s children are evicted, the parent becomes the new device leaf, so the shared prefix above survives even as divergent continuations fall away. That is the radix structure paying rent: popular stems stay, unpopular branches go.

Prefetch has its own guardrails. The storage backend parses a prefetch_threshold and a timeout policy, checked through a linear function (is_prefetch_timeout), with a configurable stop policy — the system treats “a prefetch that never completes” as an expected state with cleanup, not an exception.

vLLM’s offloading model reaches the same shape through the connector interface of Chapter 15. In offloading/common.py, a TransferJob bundles req_id with a source and destination LoadStoreSpec and is “keyed by scheduler-assigned job ID. The worker reports the job ID back when the transfer finishes.” Loads and stores accumulate separate DirectionalTransferStats — bytes, time, and a per-transfer sizes list — so dashboards can see that stores trickle in small blocks while loads arrive in large ones. The metadata also carries a jobs_to_flush set — the scheduler’s explicit cleanup list for jobs cancelled before completion, the same orphaned-work problem SGLang’s prefetch timeouts solve, expressed as a set of IDs instead of a timer.

Identity must cross machines

The local identity rules from Chapter 7 still apply: model version, exact token IDs, positions, adapter, media, cache layout, and namespace all affect whether state is reusable.

A distributed cache adds representation questions. Do all consumers use the same block size, dtype, layer layout, tensor-parallel size, and rank mapping? If not, a connector must transform the state or declare the route incompatible. A pool running TP=8 produces shards that a TP=4 consumer cannot place without either resharding logic or a hard refusal — and refusal is the cheaper correctness story, because resharding silently couples the cache to the model’s attention implementation.

Use content-derived identities or trusted metadata with collision checks. A directory entry should not become visible until the complete state is durable enough for its advertised tier. Partial writes and stale locations must fail closed: recomputation is safer than consuming incorrect KV state. An incorrect cache hit is worse than a miss in a way ordinary caches rarely are — the output is fluent text that is quietly wrong about the document. Each identity field exists because omitting it has a specific failure: drop the adapter ID and a LoRA-tuned request reads another tenant’s base-model state; drop the position scheme and attention sees correct values at wrong offsets; drop the TP size and shards simply do not line up; drop the dtype and dequantized values shift under every query. None of these fail loudly — which is exactly why the key must carry them rather than the lookup hoping.

Model updates provide a clean invalidation boundary. Include the weight version in the namespace instead of attempting to inspect whether a change happens to leave cached values equal. Comparing outputs is expensive and inconclusive; versioning the key is cheap and decisive.

Metadata and data take different paths

A cache directory answers where a prefix or block can be found. The bulk data path moves the tensor state. Keeping them separate allows small metadata updates to propagate without routing large buffers through the control service.

A remote hit becomes usable only after an ordered publication protocol. Every arrow below can fail or be cancelled independently; the protocol’s job is to make sure each failure leaves the system in the state the step before it would have left.

flowchart LR
    I["Validate semantic identity"] --> M["Resolve location metadata"]
    M --> A["Allocate destination blocks"]
    A --> T["Transfer and checksum"]
    T --> P["Publish local visibility"]
    P --> U["Use with reference or lease"]
    U --> E["Evict visibility then storage"]

Events can announce block creation, removal, or movement. Consumers need a way to handle delayed or reordered events. A location advertised moments ago may already be evicted. Treat directory results as hints until the source confirms and pins the data — concretely: the entry carries a version and the source’s answer to “do you still hold this” is the only thing that converts the hint into a transfer. Chapter 21 meets the same problem from the other side, where stale events must be discarded by generation counter rather than trusted.

The request lifecycle can look like this:

lookup -> choose source -> reserve destination -> transfer -> validate
       -> publish local mapping -> execute -> release transfer pins

Cancellation at each stage needs cleanup. A timed-out prefetch must release destination buffers and source references even if its completion arrives late. Late completions are the dangerous case precisely because they arrive after everyone stopped waiting: the cleanup must be driven by timeouts and generation counters, not by the arrival of the event that would have made cleanup easy.

Completion is a distributed fact

Both pinned implementations refuse to let one rank decide that a tier operation finished, and both pay for the agreement in different currencies.

SGLang’s writing_check counts ready write acknowledgments and then all-reduces the count across the cache group with ReduceOp.MIN — the write is considered progressed only up to what the slowest rank has confirmed. Completions drain in acknowledgment-queue order (ack_write_queue.pop(0)), so the MIN bound doubles as a safety rule: no rank processes an acknowledgment its peers have not also counted. The comment explains why every rank must enter the collective unconditionally: “ongoing_write_through can diverge across ranks (e.g. write_backup returning 0 on a subset under host memory pressure), so a conditional skip desyncs the NCCL op sequence and deadlocks under TP > 1.” This is the same lesson Chapter 14’s EPLB dummy steps taught: participation in collectives is part of correctness, independent of whether this rank has work. On the load path, loading_check pairs each completed load-back with dec_lock_ref(end_node) — the pin taken when the transfer started is released exactly when the data is confirmed, which is the lease lifetime in code.

vLLM’s offloading connector counts workers instead of reducing counts. Its OffloadingWorkerMetadata docstring specifies the rule: “The scheduler accumulates across steps and processes a transfer completion only when count reaches num_workers.” A job ID is reported {job_id: 1} by each finishing worker, aggregates sum across a step, and only a unanimous tally retires the job. Different mechanism, same semantics: a cache operation is done when every rank that owns a piece says so, and the bookkeeping exists to make partial completion visible rather than silent.

Cache-aware routing creates a trade-off

If one replica has a 30,000-token prefix and another is idle, which should serve the request? Sending it to the warm replica avoids prefill and may increase queueing. Sending it to the idle replica balances load and repeats compute.

The router should compare estimated saved work with queue and transfer cost. A static preference for the largest prefix can create a hotspot around popular state. The comparison has a natural unit — milliseconds of expected latency — and both sides convert: saved prefill from Chapter 4’s service model, queueing delay from observed queue depth times recent service time per request.

Put numbers on the opening scenario. A 30,000-token prefix saves 0.035 × 30,000 = 1,050 ms of prefill plus its 20 ms constant — call it 1,070 ms. The warm replica wins while its predicted queue delay stays under 1,070 ms, which sounds generous until the third concurrent request about the same document arrives: each queued request behind one 230 ms-class prefill adds hundreds of milliseconds, and the idle replica’s full 1,070 ms recompute starts looking cheap. The crossover is one or two concurrent requests, not ten — which is why the comparison must run per request against live queue depth, not per prefix against a static table.

The Preble paper studies distributed prompt scheduling that balances prefix reuse, load, and fairness. Its central tension is durable even as routing algorithms evolve: locality is valuable until the queue it creates costs more than recomputation.

Popular prefixes may be replicated deliberately. Replication uses more cache capacity and allows several replicas to share traffic. The control plane should measure demand before copying and remove replicas when popularity fades — replication decisions deserve the same amortization arithmetic as EPLB’s weight movement in Chapter 14, with cache bytes standing in for expert weights.

Security changes the cache key and policy

Shared state can leak information through content, timing, or existence. A tenant may infer that another tenant used a prefix by observing a faster response. Adapters or private documents can place sensitive information in KV state even if token strings are not stored beside it.

Use namespaces and access checks at lookup and transfer time. Encrypt or protect remote tiers according to the data policy. Define retention and deletion for cached state, not only original prompts. Consider whether cross-tenant sharing is permitted at all.

Randomized cache salts reduce accidental or adversarial cross-request matches, but they do not replace authorization. The safest reuse boundary is often one tenant, model version, and policy domain.

The timing channel deserves a concrete statement, because it survives every content-level defense. A cache hit on a shared prefix removes hundreds of milliseconds of prefill; the response arrives measurably earlier. A tenant that can propose prompts differing only in whether another tenant’s document is prefixed can read the hit-or-miss timing like a side channel and confirm — token by token — whether specific text exists in the shared cache. Existence leaks even when content never moves. Namespace isolation at lookup time is what closes it, which is why the check belongs in the cache itself rather than only at the API boundary.

Measure useful caching

A cache dashboard should go beyond hit rate. Track matched tokens, prefill time avoided, bytes retained by tier, bytes promoted and demoted, failed or cancelled transfers, lookup latency, eviction churn, and request latency after hits and misses.

Calculate saved compute per byte stored and per byte transferred. A high hit rate on tiny prefixes can be less useful than rare reuse of a very expensive document. Include queueing on the warm replica when evaluating the benefit. The pinned implementations make this concrete: SGLang records backup and load-back byte and token counters plus duration histograms per operation, and vLLM’s directional stats keep load and store separately — both designs distinguish the expensive direction from the cheap one, because a cache that spends 70 ms storing everything to save 110 ms once looks very different from one that spends 70 ms to save 110 ms fifty times. Lookup latency belongs on the same dashboard with its tail visible: a directory that answers in single milliseconds most of the time but stalls for seconds under load converts the cache from latency saver into latency risk exactly when traffic is heaviest.

Each metric earns its place by deciding something:

Quick reference: when to add each tier

Before investing in cache infrastructure, check whether simpler alternatives solve the problem. This table helps decide.

The general rule: local prefix caching and cache-aware routing cover the majority of reuse value with minimal complexity. Add host-memory and remote tiers only when measured cross-replica overlap justifies the publication protocol and transfer cost.

Worked example: a hit worth 110 ms

A 1-GiB prefix avoids 180 ms of prefill and takes 70 ms to load. Its gross saving is 110 ms before queueing and the opportunity cost of destination GPU memory. That value exists only if the prefix identity includes model, tokenizer, adapter, tenant namespace, token positions, and state format.

Walk the publication order and its failure points. GPU A seals the blocks — until sealed, nothing downstream may observe them. The host backup takes a read reference, copies, and publishes its location only after a checksum and generation match succeed; the directory entry that follows is a hint, not a promise. GPU B resolves the hint, obtains a lease, revalidates identity, reserves destination blocks, transfers, verifies, and only then inserts the blocks into its local index. If B cancels mid-transfer, the reservation dies unpublished and releases on the abort event — no other reader ever saw it. If B’s lookup times out instead, the outcome is a miss and a recompute, never an indefinite wait on a maybe.

Net the saving honestly: 180 − 70 = 110 ms assumes the destination had the blocks free and the source pinned them without displacing hotter state. Price the opportunity cost by what else 1 GiB of destination GPU memory would hold — about 3,300 tokens’ worth of active KV at 320 KiB per token — and subtract the queueing added on whichever replica absorbs the hit traffic. Amortization is what makes the trade clearly good: the 70 ms load cost recurs per consumer, but the 180 ms saving recurs per request, and a document queried ten times returns roughly 1.1 seconds of prefill for 70 ms of load plus the storage footprint. The same arithmetic run at hit count two is already positive; the failure case is the document asked about exactly once, where the service paid 70 ms and a gigabyte of retention to save nothing. Invalidation runs the same ladder downward: remove lookup visibility first, let existing readers finish through their references and leases, and physically delete only when the last reference drops.

Practice: write the state machine

Trace that 1-GiB prefix from GPU A through host backup, remote metadata, transfer to GPU B, and invalidation. At each transition, record identity, owner, reference or lease, checksum, timeout, and failure response.

Calculate net saved latency and saved compute per byte stored and transferred. Mark every point where completion requires agreement across ranks and say which mechanism (MIN-reduce, worker-count, or single-owner publish) you rely on. If any transition delegates correctness to “the cache,” refine it. The worked lifecycle is in Appendix G.

17. Routing, Replication, and the Control Plane

Once a service has several replicas, the frontend must decide where each request goes. Round-robin routing is attractive because it needs little state. It is also blind to the two resources that dominate inference: queued work and warm model state. The blindness is measurable with the book’s own constants. Two identical 1,000-token prompts arrive together; round robin sends one to an idle replica, where TTFT is 20 + 0.035 × 1,000 ≈ 55 ms plus one decode step, and the other behind a queued 40,000-token prompt whose prefill alone costs 20 + 0.035 × 40,000 ≈ 1,420 ms. Counts stayed balanced; time-to-first-token differed by roughly 1.4 seconds. A router that cannot see either queues or state distributes requests evenly and serves them unevenly — identical counts arrive at each replica while completion times diverge by seconds.

Routing is the cluster-level version of scheduling. The local scheduler chooses the next work on one engine. The router chooses which engine should own a new request. The same principles apply at both levels — estimate service time, admit against capacity, protect SLOs — but the router works with worse information: its view of every replica is a report about the past.

A replica has more state than “healthy”

Useful routing information can include queue length, estimated remaining work, free KV blocks, active batch composition, cached prefixes, adapters, model version, current stage role, and recent failures. Each field earns its place on the report only if some policy consumes it, and different policies lean on almost disjoint subsets:

The global router predicts a destination; the local scheduler owns execution. Telemetry flows one way, decisions the other, and the loop closes only as fast as reports travel.

flowchart LR
    A["Request and deadline"] --> R["Global router"]
    T["Stale queue and locality telemetry"] --> R
    R --> W1["Replica 1 scheduler"]
    R --> W2["Replica 2 scheduler"]
    R --> W3["Replica 3 scheduler"]
    W1 --> T
    W2 --> T
    W3 --> T

No router sees all of this perfectly. Telemetry arrives late. A decision based on an empty queue may reach the worker after several other requests. A cache entry can be evicted between lookup and assignment.

Treat routing data as a prediction. Once a worker accepts a request, it should become the authority for that request’s local lifecycle. The global router should not micromanage every token step.

The telemetry budget

Staleness has a price in requests, and Appendix A’s Little’s law prices it: at arrival rate λ and reporting delay d, the router’s picture of any queue is wrong by roughly λ·d requests. Assume a ten-replica fleet taking 40 requests per second in total — 4 per second per replica — and telemetry published every 250 ms. Each router decision is made against a snapshot that is, on average, one request out of date per replica; during a burst arriving at twice the average, two. That is the error floor under normal operation, before any network partition or slow heartbeat stretches d. It explains an empirical rule: routing improvements from better policies shrink, and can reverse, when telemetry intervals stretch past a few hundred milliseconds, because the prediction error begins to exceed the differences between destinations.

Age also determines what kind of decision an observation can still support, which is why serious routers tag every report with its timestamp and let the policy — not the data path — decide how much to trust it:

The budget cuts both ways. Publishing telemetry more often costs control-plane bandwidth and scheduler time — every report interrupts the engine step loop it describes — and finer-grained data goes staler faster because it describes a smaller, faster-changing quantity. Queue depth in requests is stable enough to report at 250 ms; free KV blocks change every admission and finish; cached- prefix inventories are large, so most deployments advertise them as compact sketches or on-change summaries rather than full listings. Match the report cadence to how fast each quantity actually moves.

Common routing policies

Round robin spreads request counts. Least-connections spreads active requests. Least-estimated-work tries to include prompt and expected output length. Session affinity keeps related turns together. Cache-aware routing values saved prefill. Priority-aware routing reserves capacity for important traffic.

A hybrid routing score compares waiting, recomputation, and movement.

flowchart TB
    C["Candidate replica"] --> Q["Estimate queue time"]
    C --> P["Estimate missing-prefix compute"]
    C --> T["Estimate transfer or adapter load"]
    Q --> S["Combined cost plus uncertainty"]
    P --> S
    T --> S
    S --> D["Choose destination and record prediction"]

Each policy sees only part of the cost. A good practical score can combine estimated queue time, execution work, cache savings, transfer cost, and a penalty for uncertain or stale telemetry.

The weights should come from measurement. A cached token has little value if the worker’s queue is several seconds long. An idle replica is less attractive if it lacks a required adapter and must load it first.

Preble studies this conflict directly: a distributed prompt scheduler must co-optimize reusable prefix state and load, because maximizing either one alone can make placement worse.

Scoring a placement

A hybrid score is just the worked example’s arithmetic generalized, term by term:

cost(R) = queue(R) + missing_tokens(R) × 0.06 ms
        + transfer_or_load(R) + risk(R)

The first two terms produce G §16’s table: R0 at 300 + 0, R1 at 0 + 240, R2 at 100 + 120. The third term catches what the simple score ignores. Give R1 a required adapter it does not hold: assume loading that adapter costs 800 ms of foreground time on first use. R1’s cost becomes 0 + 240 + 800 = 1,040 ms and it drops from second place to last — a policy that ignored the adapter term would have sent every subsequent adapter-sharing request there too, paying 800 ms each time until the adapter warmed. Transfer works the same way in disaggregated deployments: Chapter 15 priced a 6,000-token KV move at ~95 ms, which is exactly the kind of term that belongs here rather than being discovered after admission.

The risk term prices the telemetry budget from the previous dive: a destination whose last report is old, or whose locality claim comes from a sketch rather than a confirmed pin, carries a penalty proportional to how wrong it could be. In practice the penalty needs to be asymmetric. Overestimating a busy replica’s cost sends traffic somewhere slightly worse; underestimating it stacks another request onto a queue that was already the bottleneck. Set the asymmetry from measurement of your own staleness distribution, not from symmetry aesthetics.

Pricing a cached prefix

Cache-aware routing needs a price for locality, and the pinned constants give one for the worked example’s 4,000-token prefix. Its KV image weighs 4,000 × 320 KiB ≈ 1.22 GiB, so three ways of honoring a match to it differ sharply:

None dominates. Recompute burns prefill-engine time that other requests wanted; the shared copy converts a compute problem into a network problem — 10 hits per second pull 12.2 GiB/s through the holder’s links, over half of one NVLink-class link, and make that replica both popular and fragile, which is exactly G’s hot-cache hazard; replication spends HBM that otherwise holds about two-thirds of another 6,000-token conversation. The crossover variable is hit rate. At 0.1 hits per second the recompute bill is 24 ms of engine time per second — noise, and no memory or link is worth spending on it. At 10 hits per second replication buys back a full engine’s worth of prefill for a few GiB. Hit rate, however, drifts: a prefix becomes popular with a news cycle and fades with it. Replication decisions therefore need the same hysteresis as autoscaling — promote on sustained hit rate, demote on sustained absence — because a policy that flips on every crossing thrashes memory and invalidation traffic alike. Chapter 16’s invalidation ladder prices the demotion side of that flip.

Sessions need a state policy

Affinity is useful for multi-turn chat and real-time media because the worker already holds state. It also makes a worker failure or hotspot more disruptive.

Decide whether session state can migrate, be reconstructed, or is lost with the worker. A short chat prefix may be cheap to recompute. A long video session with recurrent state may need replication or checkpointing. The routing policy follows from that state policy:

Sticky routing should have an escape. If the preferred worker is overloaded or draining, the router can transfer state, recompute on another replica, or reject according to the remaining deadline. The escape condition should be evaluated per turn, not per session — a sticky session that cannot break affinity turns its worker’s queue into the user’s latency.

Global admission and backpressure

A fleet can be overloaded even when some workers still accept requests. The control plane needs a view of total queued work and stage capacity.

In a prefill/decode deployment, admission should consider both pools. Sending a request into an available prefill worker is harmful if no decode capacity will be ready afterward — Chapter 15’s coupled queues are the mechanism, and the router sits close enough to see both sides. In an MoE deployment, a network or expert hotspot can limit capacity while aggregate GPU utilization looks low; Chapter 14’s balancedness statistic is the sort of signal that distinguishes “spare capacity” from “capacity gated by one rank.”

Global admission can reserve capacity, reject work that cannot meet its SLO, or return a retry delay. Backpressure should reach the original caller or durable upstream queue. Uncoordinated retries multiply load precisely when the service has the least spare capacity — a retry storm converts a 30-second overload into a five-minute one, because every timed-out client re-submits exactly when workers are least able to help.

Admission is routing with a veto

The hybrid score ranks destinations; admission decides whether the request may go anywhere, and the Atlas SLO turns that veto into arithmetic. Suppose the request arrives at the router having already spent 350 ms in an upstream gateway. TTFT must stay within 600 ms, so the remaining budget is 250 ms. Score the same three replicas against it: R1 survives with 10 ms of margin (its cost is 240 ms) and R2 with 30 ms (220 ms), while R0’s 300 ms makes it inadmissible outright. Tighten the upstream spend to 400 ms and the budget falls to 200 ms: no destination qualifies. The correct action flips from routing to rejecting — return a retry delay or shed — before any worker sees the request.

That flip is worth internalizing because admitting into certain violation is not neutral. An admitted request that misses its deadline still consumes its prefill and decode slots, pushing neighbors’ queues longer and converting one missed SLO into several. Early rejection spends nothing but the caller’s patience, and the retry-delay estimate comes from the same score: if R2’s cost is 220 ms now, a delay of a few hundred milliseconds plausibly restores admissibility, whereas a blind immediate retry lands on unchanged queues. The veto also protects the deadline term from being gamed implicitly — a score without admission quietly routes deadline-starved requests to whichever replica misses by the least.

Autoscaling has memory

Traditional autoscaling often reacts to CPU utilization or request count. An inference replica has long startup stages: image pull, model load, distributed initialization, compilation, graph capture, and cache warm-up. By the time a new replica is ready, the original burst may be over.

Useful signals include queueing delay, SLO headroom, estimated work, KV pressure, stage imbalance, and sustained arrival trends. Scaling policy needs hysteresis so the fleet does not repeatedly add and remove replicas around one threshold.

Scale-down also costs state. Draining a warm replica can discard valuable prefixes or sessions. Compare the saved capacity cost with the future cold penalty. A minimum warm pool may be cheaper than scale-to-zero for latency- sensitive models.

Startup latency versus reaction time

Assume the pinned models’ startup sequence lands at four minutes end to end — image pull tens of seconds, weight load dominating, then compilation and graph capture from Chapters 8–9 adding minutes before the first useful step, then cache warm-up. A reactive autoscaler that triggers when utilization crosses a threshold for two consecutive windows adds capacity four minutes after the signal. Any burst shorter than that pays nothing and costs a replica; any burst longer gets relief only for its tail. This is why inference scaling signals lead rather than lag: queueing delay and arrival trend predict the need roughly one startup-duration ahead, and the scale-out decision is really a forecast with a four-minute horizon.

Hysteresis sizing follows from the same number. The down-scale threshold should sit far enough below the up-scale threshold that normal traffic noise cannot traverse both within one drain cycle, and drained replicas should release state in the order Chapter 16’s invalidation ladder defines — visibility first, storage later — so a flapping fleet does not thrash the distributed cache along with itself.

Membership and deployment

When a replica joins, the router must not send traffic until weights, parallel groups, graphs, and health checks are ready. When it leaves, new traffic should stop before current work drains. Forced termination needs a retry and state-loss policy.

Rolling out a new model version creates two cache namespaces and possibly two sets of compiled artifacts. Requests in a session should not cross versions accidentally. Canary routing must compare equivalent traffic and keep the old version available for rollback.

The namespaces interact with locality in a way that is easy to miss at rollout time. A hot prefix replicated across the fleet before the rollout — per the pricing dive above — is warm only in v1’s namespace; a canary replica holding v2 weights cannot read those blocks, so every canary request pays full recompute and looks artificially slow against v1 traffic that rides warm caches. The comparison is not measuring model quality; it is measuring cache temperature. Fair canaries either exclude cached-prefix advantages from both sides, route each version its own warm-up period, or compare only on prompts known to be cold for both namespaces.

Membership changes are distributed events. Use generations or epochs so a delayed health message from an old process cannot make a dead replica current again. The failure this prevents has a specific shape: replica X crashes, its replacement registers, then a queued health report from X arrives and a router that keys on content rather than epoch marks both alive and splits traffic to a process that no longer exists. An epoch number on every membership message makes the stale report self-invalidating.

Failure changes routing cost

A timeout can mean a slow request, a failed worker, a partitioned network, or an overloaded dependency. Retrying on another replica may recover and duplicate expensive work. Hedging can reduce tail latency while consuming extra capacity.

Requests should carry stable IDs and attempts. Output protocols need a rule for which attempt is authoritative. State transfers and cache writes should be idempotent or safely abandoned. The router should open a circuit around a failing destination rather than continuing to discover the same failure per request:

The half-open probe matters for inference specifically because recovery is not instantaneous: a replica that just restarted must reload weights, rebuild graphs, and warm caches before it serves honestly. A probe that succeeds at health-endpoint level while the first real request still pays cold-start costs will flap the circuit. Gate the transition back to closed on the same readiness signals the membership protocol used at join time.

Hedging has a computable break-even even without a distribution handy. Assume 1 percent of requests strand on a slow path costing 3 extra seconds. Hedging all of them after 500 ms spends one duplicated request — including its prefill, which Chapter 4 prices from prompt length — per hundred requests, to remove up to 2.5 seconds from 1 percent of tails. Whether that trade is good depends on what a duplicated prefill displaces at the second replica, which is why hedge decisions belong in the same scoring framework as everything else in this chapter rather than in a static percentage.

Worked example: partial locality wins

R0 has all 4,000 reusable tokens and 300 ms of queue. R1 is idle with no match. R2 has 2,000 matched tokens and 100 ms of queue. If recomputation costs 0.06 ms per missing token, estimated placement costs are 300, 240, and 220 ms. R2 wins.

Walk each term. R0: perfect locality, zero prefill, but the request inherits every queued request ahead of it — 300 ms of other people’s work. R1: zero queue, but 4,000 missing tokens at 0.06 ms each = 240 ms of prefill. R2: 2,000 matched tokens leave 2,000 to compute = 120 ms, behind 100 ms of queue; 100 + 120 = 220 ms, twenty milliseconds under R1 and eighty under R0. The margin is thin — which is the point. Shift R2’s queue to 150 ms and R1 wins; shift R0’s queue to 250 ms and R0 wins. Partial-locality routing earns its complexity only because these quantities move on the scale of single requests, and the score re-evaluates per arrival.

Cache-only routing chooses R0; least-queue chooses R1. A hybrid cost makes its assumptions visible and can add transfer, adapter load, stale-telemetry risk, and deadline penalty. It remains a prediction, so the worker reports the actual queue and match observed at admission — closing the loop and giving the next revision of weights its training data.

Practice: simulate a hot prefix

Implement the three-replica case, then add bursty arrivals, delayed telemetry, finite caches, and one increasingly popular prefix. Compare round robin, least-work, cache-only, and the hybrid estimate.

Plot goodput, queue percentiles, recomputed tokens, occupancy, and imbalance. Find when replication of the hot prefix repays its memory and add hysteresis to prevent oscillation. Delay telemetry deliberately in the simulator — G’s note is worth keeping: perfect instantaneous queue knowledge would make the router unrealistically powerful. The worked decision is in Appendix G.

The simulation completes the path from one request to a distributed text service. Part IV applies the same ideas to models whose serving loops are not limited to text decode.

Part IV — Beyond Text-Only Decoding

The same state-and-topology reasoning applied where the decoder loop does not rule: images that flow through encoders, diffusion pipelines that iterate over a latent, learning loops that rewrite the model underneath a live service, and interactive sessions, long reasoning traces, and agentic waits that must answer within a human sense of time.

Chapters 18–21

18. Multimodal, Encoder, and Pooling Workloads

A user uploads a 20-second video and asks one short question. The language model may generate only ten tokens, yet the request can be far more expensive than a long text prompt. Before the first output token appears, the service must fetch and validate the media, decode it, sample frames, resize them, run a vision encoder, project the results into the language model’s representation, and merge them with the text prompt under a template that both sides agree on. Each stage has its own hardware profile, its own queue, and its own notion of identity.

If you measure only language decode, you miss most of the request. A serving system that treats “the prompt” as an opaque token array will discover the pipeline the hard way: CPU saturation from video decoding, an accelerator idle while encoders wait behind preprocessing, and cache hits that never happen because two stages disagree about what makes two pieces of media “the same.”

Media begins as untrusted bytes

Images, audio, video, and documents arrive in formats optimized for storage and transport. Their decoded representation can be much larger: a few megabytes of compressed video may expand into hundreds of full-resolution frames, and a document upload may contain many high-resolution pages. Expansion is the attack surface as well as the cost. Decompression bombs, malformed codec streams, and metadata that lies about dimensions all present themselves as ordinary requests.

A multimodal request is a pipeline before language decoding begins.

flowchart LR
    B["Media bytes"] --> D["Decode and validate"]
    D --> P["Resize and preprocess"]
    P --> E["Modality encoder"]
    E --> X["Projected feature tokens"]
    T["Text tokens"] --> L["Language prefill"]
    X --> L
    L --> O["Autoregressive output"]

The frontend should validate type, byte size, dimensions, duration, frame count, and decompression limits before expensive work begins — ideally before the bytes are fully buffered, since a validation failure should never have paid for a decode. Fetching remote media needs timeouts, address restrictions, and a policy for redirects, because a URL field is otherwise a tool for making your cluster fetch arbitrary network content. Media parsers and codecs belong inside the service’s security boundary, not in the API server process.

Preprocessing then converts the input into the exact form expected by the model: resizing, normalization, frame sampling, audio resampling, or document layout processing. These choices affect model output — two frames sampled from the same video at different rates produce different answers — so processor version is part of the execution identity and every downstream cache key. Chapter 6’s execution request concept extends naturally: the identity of a multimodal execution request includes the preprocessing configuration that produced its tensors.

Where the decode actually happens

Frame sampling looks like a preprocessing detail and is often the single largest cost multiplier in the pipeline. Take the chapter’s opening request: a 20-second video at 30 frames per second contains 600 frames, but a typical vision-language configuration samples one to two frames per second — call it 20 frames consumed. A naive pipeline decodes all 600 frames and discards 580: thirty times the necessary codec work, paid on the CPU, before the encoder sees anything. Seek-based sampling instead jumps directly to each sampled timestamp — except codecs do not store arbitrary frames; decoders must start from the previous keyframe, so the cost of sampling timestamp t depends on how far t sits from keyframe boundaries in the container’s group of pictures. Two videos of identical duration and resolution can differ by several times in decode cost purely through keyframe spacing — an identity- irrelevant property that never appears in any cache key yet dominates the stage the decoded-media tier is supposed to save. Practical services respond by keeping decode off the accelerator entirely, bounding concurrent decodes, and treating decode throughput as a first-class capacity number rather than an implementation accident.

The encoder is a separate workload

A vision or audio encoder usually processes many positions in parallel. Its shapes depend on resolution, patch count, frame count, or audio duration rather than on a token count chosen by the caller. It may be compute-heavy while language decode is memory-bound — the opposite hardware profile — which is why co-locating them on one GPU works at low load and fights itself at high load.

The output is a tensor of media features. A projection layer maps those features to the language model’s representation, and placeholder positions tell the language model where they belong:

media -> decode and normalize -> encoder -> projected features
                                                   |
text -> template -> tokenizer ---------------------+
                                                   v
                                     language prefill -> decode

The placeholder and feature lengths must agree. Truncating a text prompt can accidentally remove media positions — the placeholders live in the token stream, so a length-based truncation written for text silently corrupts multimodal requests by cutting feature anchors while leaving the features themselves intact and unused. Batching requests with different numbers of images or frames requires ragged metadata: per-request lists of grid shapes, feature lengths, and offsets. These are correctness concerns, not only tensor-shape concerns — a mismatched length usually produces plausible garbage rather than an error.

The feature length itself is computed, not read off the request: images derive it from their patch grid after any token-merging the model applies, audio from sample counts and stride, video from frames times per-frame grids. That derived length must travel with the features as metadata — grid shapes, per-modality feature lengths, original image sizes — because the language worker’s job at merge time is to splice feature tensors into placeholder positions inside the token stream, and it can only do that if the geometry arrives alongside the bytes. SGLang’s receiver carries exactly this cargo per part (img_grid_thw, video_grid_thw, audio_feature_lens, plus model-specific video attributes) and reassembles it when parts complete; vLLM’s connector keys saved tensors by mm_hash for the same reason. A feature tensor without its geometry is not a cacheable object at all — it cannot be placed in a prompt.

Batch compatible encoder work

Encoders benefit from batching, but compatible shapes matter more than batch size. Padding every image to the largest resolution in a batch repeats Chapter 13’s capacity-padding trap in a sharper form, because vision patch grids scale with area: an image with twice the side length of another occupies four times the patches. Pad three quarter-resolution images up to one full-resolution neighbor and you pay roughly four times their necessary encoder compute for the privilege of batching them — a loss no reasonable batch-size win repays.

Bucketing by resolution, aspect ratio, frame count, or audio length improves efficiency while adding queueing delay: an image that could run now waits for company in its bucket. The right window depends on the endpoint. An offline document-indexing job can hold a bucket for a couple of hundred milliseconds to fill a large batch — assume its TTFT budget is minutes, not milliseconds. An interactive visual question should not wait long for another image of the same size — its user is watching, and against this chapter’s 465 ms worked trace, even a 50 ms bucket wait is a tenth of the entire budget.

Padding is the same trap twice

The mechanism deserves the comparison spelled out. In MoE dispatch, padding to capacity wasted FLOPs on expert slots that computed nothing. In encoder batching, padding to max shape wastes FLOPs on real transformer work over empty patches — worse, because the padded compute is numerically meaningful and therefore cannot be skipped by clever kernels; masked attention over pad patches still costs memory traffic. And there is a second layer: ragged batches also pad in the time dimension when a scheduler holds a fast request for a slow bucket mate, which is latency padding — invisible in throughput dashboards and fully visible in TTFT percentiles. Both paddings respond to the same medicine: schedule compatible work together and incompatible work immediately, and let the bucket boundaries come from measured shape distributions rather than round-number defaults. Measure preprocessing and encoder queueing separately either way — a busy CPU decoder can starve an otherwise idle accelerator, and moving transforms onto the GPU trades that starvation for contention with model execution plus extra copies across the PCIe boundary.

Encoder outputs can be reused

Users often ask several questions about the same image, document, or video. Reusing encoder features avoids repeated media work, and the cache key must include content, preprocessing configuration, encoder and projection weights, precision, and any model-specific placeholder layout. Miss any term and two different computations collide under one key; include redundant terms and hit rates collapse. Caching processed bytes alone saves decoding. Caching encoder outputs saves more compute and consumes more space. Caching language KV state after the first question can save still more, but may be tied to the exact conversation template. These are distinct cache layers with different reuse scopes — the chapter’s second diagram — and they fail independently.

Reuse can occur at several boundaries with different identity rules.

flowchart TB
    M["Stable media identity"] --> C1["Decoded-media cache"]
    C1 --> C2["Preprocessed-tensor cache"]
    C2 --> C3["Encoder-output cache"]
    C3 --> C4["Language-prefix KV cache"]
    C4 --> Q["New question about same media"]

Treat privacy carefully. Encoder features can reveal information about the original media — they are sufficient to reconstruct coarse image content in known attacks — so the same tenant, retention, and deletion policy used for source content must reach into every tier that derived from it.

The tiers also have a natural eviction order, and it is worth making explicit. Encoder outputs are the largest per-item objects and the most narrowly valid — bound to encoder weights, projection weights, precision, and preprocessing version — so they should evict first among the reusable tiers. Decoded pixels are smaller and survive model upgrades: a new vision tower still consumes the same decoded frame, so the pixel tier outlives every weight-dependent tier above it. Language KV sits under Chapter 16’s policies entirely. A model-version bump therefore cascades upward through the diagram: KV entries die, encoder outputs die, preprocessed tensors die if normalization changed, decoded pixels survive. Deploying that cascade as one invalidation event — rather than letting stale tiers answer under new versions — is Chapter 16’s identity discipline applied to one more cache family.

Inside SGLang’s encoder cache

SGLang’s disaggregated encode path implements exactly this tier, and reading it at the pinned SHA shows how many distributed-systems problems hide inside “just cache the features.” The encoder lives in encode_server.py, and its encode_with_global_cache runs each request through a three-outcome per-item pipeline: items either hit the global cache, miss and encode now, or fall back — nominally hits whose prefetched data failed to arrive, re-encoded rather than awaited. The fallback path exists because a hit is a claim about remote state, and claims need deadlines: _wait_global_cache_prefetch waits on check_prefetch_progress under a 60-second timeout, and any item still absent goes to fallback_indices with the log line “cache-hit items failed to load, falling back to ViT.” A cache that trusts its own hit mask hangs requests.

Identity has teeth here. Hashes are computed per grid entry — one per encoder grid cell — and the comment on the length check explains why: “a leaf-space list would size-mismatch rank>0’s mask (zeros(num_items)) and deadlock TP.” Rank 0 computes hashes and queries batch_is_exist; every other rank allocates a zero mask and joins the broadcast anyway, because the lookup result crosses the TP group as a collective and a rank that skips it desyncs the op sequence — the same participation-as-correctness principle as Chapter 14’s dummy steps and Chapter 16’s MIN-all-reduce. Note also who writes: only rank 0 stores to the pool and assembles the response embedding; other ranks return (0, 0, 0, None, None). Single-writer avoids N copies of the same insert racing through the pool.

The final ordering is the surprising one: cache insertion happens after the response is assembled, in a fire-and-forget task — store_to_pool_async hands back device-to-host handles, and _launch_global_cache_insert waits on them in asyncio.to_thread inside _background_insert. The request never pays for its own cache write; a following request does. If the process dies between response and insert, the recompute simply happens again — an acceptable loss priced against putting a D2H copy on the interactive path.

Apply stage disaggregation to encoder work

Chapter 15 owns the connector protocol, coupled-queue arithmetic, and failure state machine for encoder/prefill/decode separation. Multimodal serving contributes the request-specific inputs to that decision: decoded-media cost, feature size, encoder-cache identity, reuse frequency, and the current encoder and language-worker queue ages.

Short, uncached media often stays colocated because the transfer boundary is pure overhead. Long videos, repeated questions over one image, and independently scalable encoder bursts can repay a separate pool. Route per request from measured encoder time, transfer time, cache state, and queue age; do not repeat the generic pool-sizing derivation here.

The official vLLM disaggregated-encoder guide shows the encoder-to-language connector boundary and cross-process feature reuse. Treat its support matrix as release-specific; the cost and ownership test above is the durable decision.

Not every encoder leads to generation

Embedding, classification, reranking, and reward endpoints produce a complete output after one model pass — no decode loop, no KV cache to manage, a genuinely different serving regime despite sharing the encoder machinery. Their main questions are dynamic batching, padding, pooling, output normalization, and latency limits.

Pooling endpoints finish after projection and must restore item order.

flowchart LR
    I["Variable-length inputs"] --> B["Shape-aware batch"]
    B --> E["Encoder"]
    E --> P["Pooling or token projection"]
    P --> O["Vectors, labels, or scores"]
    O --> R["Restore request and item order"]

Reranking shows the regime’s characteristic hazard: one query paired with hundreds of documents. Flatten pairs into a batch and scores must return to the correct request-and-document order; let one wide query-document pair dominate the batch and everyone else’s latency follows the largest pair. Limit work per request and restore order by carrying indices through the batch, not by trusting arrival order.

Embedding APIs need a precise normalization and truncation contract — whether “truncate” means dropping tokens from the right or pooling early changes downstream similarity values. Reward models may return one scalar per candidate or token-level values. Using a generation-oriented output path without defining these semantics creates subtle compatibility errors that surface only when a client switches providers.

Padding is endpoint-visible accounting, not an implementation footnote. A mean-pooling model must exclude padded positions; a last-token model must agree whether “last” means the final non-padding token; a normalized embedding API must define whether normalization happens before or after truncation. Rerankers must carry both request and document indices through flattening, batching, and unflattening so that a scheduler reorder cannot silently permute scores.

The supported model families and task mappings evolve, so use the current vLLM pooling-model documentation and SGLang embedding API as implementation references rather than treating a generation endpoint with decode disabled as the specification. A conformance fixture should pin pooling method, normalization, truncation side, padding behavior, output shape, and item order. Those six fields are the portable contract.

Worked example: which cache tier matters?

An image request spends 35 ms receiving and fetching, 28 ms decoding and preprocessing, 40 ms in the encoder queue, 115 ms in the vision encoder, 12 ms transferring features, 190 ms in language queue and prefill, and 45 ms to the first token. The stages sum in order: 35 + 28 = 63, + 40 = 103, + 115 = 218,

• 12 = 230, + 190 = 420, + 45 = 465 ms of TTFT.

Walk the tiers against that trace. A processed-image cache saves only the 28 ms decode-and-preprocess stage — real, but a 6 percent improvement bought with a pixel-store infrastructure. An encoder-output cache saves preprocessing, encoder queue, and encoder: 28 + 40 + 115 = 183 ms, taking TTFT to 465 − 183 = 282 ms — nearly half off — at the price of retaining a larger, version-specific tensor per image. Full language-prefix reuse would also erase most of the 190 ms language stage, but it is legal only when the entire earlier conversation — including the media tokens’ placeholders — is an unchanged prefix of the new request; a different question about the same image breaks the match at the first new token, which is why the encoder-output tier is the one that survives follow-up questions.

Now the second question, same image, different wording. With the encoder-output cache warm it still fetches (35), transfers (12), queues and prefills (190), and decodes one token (45): 35 + 12 + 190 + 45 = 282 ms — exactly the cold trace minus the saved 183. Adding a decoded-media hit removes the fetch too: 282 − 35 = 247 ms. The identity requirements are exactly G’s list — same media identity, preprocessing configuration, model version, feature layout — and the acceptance test is empirical: compare output against a cache-disabled request. Any drift means an identity term was missed, most often precision or processor version.

Disaggregation enters as a swap inside the same arithmetic: moving encoding off the language workers replaces the 12 ms feature transfer with 35 ms but removes an 80 ms queue at the language worker, netting −57 ms — worthwhile for cached or heavy media, and a net loss for uncached tiny images where the extra boundary is pure overhead. The decision is per-request and scoreable, not architectural.

Practice: profile two questions about one image

Use the timings above for the first question, then model a second question with the same image. Compare no cache, processed-media cache, encoder-output cache, and legal language-prefix reuse. Record identity requirements, bytes retained, latency saved per byte, and cache-disabled output equivalence.

Then decide whether a remote encoder with 35 ms feature transfer is worthwhile if it removes 80 ms of language-worker queue. See Appendix G.

19. Diffusion, Image, Video, and World Models

A text-to-image request does not append one pixel at a time. It starts with a noisy latent representation and repeatedly transforms that latent toward an image. A text-to-video model may perform the same loop over a large spatiotemporal tensor. Where language serving grows its output one token per engine step, a diffusion request holds its entire output in memory from the first step and refines it wholesale — the unit of progress is a full-image update, and the unit of cost is that update repeated dozens of times.

The serving system still needs batching, parallelism, caching, graphs, and stage placement—but their meaning changes. Batching keys on shapes that the caller implies through resolution rather than token count. Caching trades approximation for compute inside a single request rather than exact reuse across requests. Graph capture keys on the request’s geometry. And the loop’s rigid structure — same shapes every step — makes diffusion, oddly, an easier target for graphs than language decode.

Follow the diffusion pipeline

A common pipeline contains four stages:

Diffusion repeats a denoising stage around persistent latent state.

flowchart LR
    P["Prompt"] --> T["Text encoder"]
    T --> C["Conditioning"]
    N["Initial noise"] --> D["Denoising model"]
    C --> D
    D --> S["Step scheduler"]
    S -->|next latent and timestep| D
    S -->|final latent| V["Image or video decoder"]
    V --> O["Media output"]

1. a text encoder turns the prompt into conditioning features;
2. a scheduler defines a sequence of noise levels or timesteps;
3. a denoising network, often a diffusion transformer, updates the latent at each step;
4. a decoder converts the final latent to pixels or frames.

Safety checks, upscaling, interpolation, or audio generation may add more stages. The denoiser usually dominates compute because it runs many times: in the chapter’s worked trace, thirty steps at 24 ms contribute 720 of 815 total milliseconds — 88 percent — so every optimization in this chapter is, one way or another, an attack on that repeated stage.

Unlike LLM decode, the latent keeps a stable shape during the loop. Every step advances a full image or video representation: same tensor, same attention pattern, same kernel launches, step after step. This makes graph capture and shape-compatible batching attractive in a way language decode is not — a language engine step changes shape as sequences finish and prefill interleaves, while a diffusion step is the same computation with different numbers in it.

The step is also the natural deadline quantum. A video model producing frames progressively exposes intermediate latents that are already watchable; a request’s perceived latency tracks when usable frames emerge, not when the final step lands. Schedulers that treat the whole pipeline as one opaque request throw that visibility away — but exploiting it has a price worth stating. Every mid-stream preview must pass through the 55 ms latent decoder, so a service that renders previews at steps 10 and 20 spends two extra decoder passes per request: declared as 110 ms of added device work against a perceived latency win of hundreds of milliseconds for the first glimpse. The exchange can be favorable and still should be made deliberately — cap preview count, decode at reduced resolution where the VAE allows it, and let the client’s viewport decide whether an early look is worth the capacity.

Compatible requests can share a batch

Two requests can batch when the active pipeline stage and tensor shapes agree. For image generation, resolution, latent channels, timestep, guidance mode, model variant, and backend may all matter. Video adds frame count and temporal layout. The compatibility set is bigger than language’s (sequence length plus model version) because the denoiser’s work items are full tensors rather than token rows, and because guidance doubles the branch structure — a classifier-free-guidance request may want both its branches batched together, or each branch batched across requests, and the two choices conflict.

A serving scheduler can wait briefly for compatible work, merge it, and split outputs afterward. The waiting window has a computable break-even. Assume two same-resolution requests arrive close together, and batching two denoisers raises per-step cost from 24 ms to a declared 31 ms. Run separately, back to back, the pair completes at 815 + 815 = 1,630 ms; run batched from the start, both finish near 30 × 31 = 930 ms plus one pipeline’s fixed stages. The gap — roughly 700 ms of machine time — funds a wait: even if the partner arrives 200 ms late, batching beats queuing whenever the wait is shorter than the partner’s entire serialized service time. The window should be sized in those terms (a fraction of one pipeline duration), not in round milliseconds. Padding a short video to match a long one may waste too much compute — the same pad-to-max trap as Chapters 13 and 17, now in the temporal dimension.

Earlier advice often treated video generation as a batch-size-one workload. That remains reasonable for very large, latency-focused generations, but it is not a law. SGLang’s current diffusion documentation includes compatible-request inference batching for image and video pipelines. The benefit depends on workload and stage.

Signatures decide replay

Graph capture meets this compatibility problem head-on, and SGLang’s “breakable CUDA graph” runner — the diffusion-side wrapper in breakable_cuda_graph/runner.py — is a compact study in making replay safe. Its call contract is the one Chapter 9 argued for, enforced: “Capture is an explicit, idempotent capture() call (driven at warmup) so that serving never triggers a fresh capture.” A call either finds a captured graph for its signature or runs eagerly.

The signature is the interesting part. Tensor leaves key on shape and dtype so values may change per replay; non-tensor constants must join the key because they are “baked into the captured Python control flow”; mutable objects key on identity, “to avoid replaying a graph whose eager [execution would differ].” Miss the key and the runner does not fail silently — it emits a one-shot diagnostic listing the differing fields, ending in a hint that deserves framing: “graphs replay only for the exact shapes captured at warmup … the auto-derived warmup resolution is the model default, which can differ from the resolutions you actually serve — declare every served resolution explicitly.” A graph-bucket misconfiguration announces itself as eager fallback with a warning, which is why the bucket set must come from served resolutions, not defaults.

Replay itself is three moves with one subtlety: copy live inputs into the captured static buffers (buf.copy_(live, non_blocking=True)), replay under a token scope, then clone the output — because the other CFG branch “shares this static output buffer when shapes match,” and the caller may still be holding the conditional branch’s result when the unconditional branch replays. “The clone is one cheap DtoD copy relative to the full DiT” — the price of static-buffer reuse, charged exactly where aliasing would corrupt results. And when the structure changes under a matching key — “should not happen” — the runner falls back to eager “rather than copy mismatched buffers.” Every fallback path chooses correctness over performance without being asked twice.

Caching trades quality for skipped work

Nearby denoising steps produce similar intermediate features. Cache methods reuse selected block outputs instead of recomputing the entire network. DeepCache studies reuse of high-level U-Net features across steps; TeaCache uses timestep-aware differences to decide when cached outputs can be reused in video diffusion. The two mark the design space’s ends:

Serving optimizations act at different boundaries in the loop.

flowchart TB
    R["Request resolution, steps, and conditioning"] --> B["Compatible batching"]
    B --> G["Graph bucket"]
    G --> K["Cross-step cache policy"]
    K --> P["Parallel or staged placement"]
    P --> Q["Latency and visual-quality evaluation"]

Fixed schedules are simple to reason about and audit; signal-driven skips concentrate the savings where the trajectory is actually redundant. Production systems increasingly want the second property but must then trust a runtime measurement — which is why the decision’s internals matter, not just its hit rate.

Unlike exact KV reuse for an identical language prefix, diffusion feature caching is often approximate. Skipping work can change visual quality. The cache policy needs an error or quality budget, and evaluation should cover motion, prompt adherence, temporal consistency, and artifacts—not only latency. The best cache interval may vary over the denoising trajectory — some steps are more sensitive than others — so a fixed “reuse every N steps” rule is a baseline, not a policy.

Inside TeaCache’s skip decision

SGLang’s integration in cache/teacache.py makes the approximate-reuse contract concrete. The signal is the modulated input — the transformer’s input after timestep conditioning — and the test is a relative L1 distance between this step’s and the previous step’s modulated inputs: diff.abs().mean() / prev.abs().mean(). That raw distance passes through a per-model polynomial rescale (np.poly1d(coefficients)) before it accumulates — the decision variable is the running sum since the last real compute, not this step’s difference alone. When the accumulator crosses teacache_thresh, the step recomputes and the accumulator resets; until then, the block reuses its cached residual.

Three details carry the systems lessons. First, boundary steps always compute (is_boundary_step forces should_calc), because the trajectory’s endpoints shape everything downstream. Second, the check fails open: “Defensive check: if previous input is not set, force calculation” — an uninitialized cache costs a compute step, never a corrupted image. Third, classifier-free guidance gets separate positive and negative caches, and models whose CFG structure the method has not validated “auto-disable TeaCache when CFG is enabled” — an explicit compatibility gate rather than a silent wrong-answer risk, the same philosophy as Chapter 10’s quantization fallbacks.

There is also a cost the paper view omits: the decision reads .cpu().item() — a device-to-host synchronization every step, inside the loop graph capture would otherwise own. Approximate caching and graph replay are not free complements; the skip decision buys 24 ms of denoiser work by spending a sync, and a deployment should measure the exchange rather than enable both and assume addition. Against the worked trace, skipping ten of thirty steps saves at most 240 ms — the accumulator design exists precisely to make those ten the safe ten.

The accumulator’s shape is also worth internalizing, because it explains the skip pattern. To make it concrete with declared numbers: suppose the rescaled per-step distances hover around 0.02 in a quiet stretch and 0.06 when the image is changing fast, with a threshold of 0.15. The quiet stretch skips seven consecutive steps (7 × 0.02 = 0.14, still under threshold) before one more forces compute; the busy stretch skips only two. Savings therefore cluster where the generation is stable — early background refinement, static regions of a video — and vanish exactly where quality risk is highest. That self-alignment is the argument for signal-driven policies over fixed intervals, and it is why hit rate alone is a misleading metric: two policies with equal skip counts can land their skips in entirely different places on the trajectory.

Parallel dimensions follow the latent

Tensor parallelism can split model weights. Sequence parallelism can divide spatial or temporal tokens. Ring and Ulysses-style attention move or transpose state so each rank handles part of a long media sequence. Classifier-free guidance can place conditional and unconditional branches on different ranks.

These dimensions can compose, but each introduces a collective or transfer. Video shapes are large enough that context-parallel communication can dominate. Map the logical partition to the fastest links and include decoder or VAE memory in the plan.

For one pipeline, draw a tensor shape at every stage and step. Mark which dimension each parallel method splits. If two methods split the same dimension in incompatible ways, their size arguments may look valid while the execution plan is not — the composition conflict is invisible in per-method arithmetic and obvious in the annotated shape table.

Composing splits on one tensor

The annotated-shape exercise deserves its concrete form. Take the video latent from the stage-split estimate — frames F, height H, width W, channels C, plus attention heads D — and watch where each method wants to cut:

Composition works while each claimant takes a different axis: CFG on the branch dimension, ring attention over F, sequence parallelism over H·W, tensor parallelism over channels. The plan breaks when two methods claim one axis — Ulysses-style attention partitions heads, which collides with a tensor-parallel scheme that already divided them, leaving some ranks no work while others carry two roles’ communication. And every added split multiplies the collectives that must include every rank — Chapter 14’s participation rule again: a diffusion step is only as fast as its slowest collective member, and a composition whose per-step exchange exceeds the 24 ms step budget converts parallelism into overhead. Price the composition per step, not per method.

Stages can use different workers

The text encoder, denoiser, and image or video decoder have different compute and memory profiles. A disaggregated pipeline can scale them independently and assign different accelerators. Intermediate tensors must move between pools, and their sizes span orders of magnitude. Assume a common VAE that downsamples eight times spatially and four times temporally with sixteen latent channels: a 1080p, 24-frame clip arrives at the decoder boundary as roughly 240 × 135 × 6 × 16 values — about 6 MiB in BF16 — while its text conditioning is a few hundred kilobytes. The text-embedding boundary is nearly free to cross; the latent-to-decoder boundary is not, and it lands on the request’s critical path with no compute hiding it.

Stage separation is most attractive when one stage is reused, batched differently, or strongly imbalanced — a safety checker fanning out over finished latents, or a text encoder whose queue fills with short prompts while the denoiser runs seconds per request.

Note how small the pure-transfer term is at image scale. The 6 MiB decoder latent crosses the declared NVLink-class link in a + S/b = 20 µs + 6 MiB ÷ 450 GB/s ≈ 34 µs — about a seventh of a percent of one denoising step — and even a slower storage-class link prices it under a millisecond. The boundary’s real cost is serialization: the request now visits two queues and cannot overlap its decoder wait with anything, so the split pays only through the utilization it buys, exactly as G’s guidance says. Bytes alone almost never settle it. SGLang’s pinned source implements diffusion parallel groups, compatible batching, graph runners, caching integrations, and stage disaggregation under multimodal_gen/runtime — including a dedicated disaggregation package with dispatch policies, roles, and an orchestrator, the E/P/D pattern of Chapter 15 re-derived for diffusion stages. The code illustrates how text-generation engine ideas can be adapted without pretending the loops are identical.

Real-time video keeps a session alive

World models and causal video generators may accept new observations while producing future frames. The service now owns a long-lived session with recurrent or attention state. A user may steer the scene, interrupt generation, or change controls.

The scheduler must meet per-frame deadlines, not merely finish a request. At a declared 24 fps the per-frame budget is roughly 42 ms — tighter than the book’s ITL SLO — and it recurs forever; a session that misses one deadline has not queued work, it has dropped frames a human sees. State may need to migrate when a worker drains. Dropping quality, resolution, frame rate, or lookahead can be a graceful overload response: cutting lookahead by one chunk refunds one chunk-period of slack immediately, which is why lookahead is the first knob to reach for and resolution — which changes graph buckets and batch compatibility — the last.

Batching live sessions is possible when their frame clocks and shapes align. One late session should not stall all others, so the batch may need deadlines or selective dropping.

Sessions are state machines

The pinned source’s realtime/session.py shows how little machinery this takes when the invariants are strict. Sessions live in an LRU (OrderedDict, max_sessions=64); each chunk attaches to its session by id and the cache bounds memory by evicting the least recently touched. The protocol lives in one integer: the request’s block_idx. A chunk arriving with block_idx > 0 whose session state is missing raises immediately — “Missing realtime session state” — because a mid-stream chunk after eviction would otherwise silently fabricate a fresh session and corrupt the stream’s continuity. A chunk with block_idx == 0 is an epoch boundary: the client is restarting the stream, so the cache disposes the old state and installs the new. That is Chapter 17’s membership-epoch discipline in miniature — the stream’s first block is a self-declaring generation marker, and everything after it demands proof the generation is still alive. Eviction disposes state explicitly (_dispose_session), swallowing and logging disposal failures rather than letting one bad teardown poison the cache — the same defensive shape as every other state-owner in Part III.

Worked example: 30 repeated steps

An image pipeline uses 18 ms for text encoding, 30 denoising steps at 24 ms each, 55 ms for latent decoding, and 22 ms for postprocessing. The timeline sums as 18 + 720 + 55 + 22 = 815 ms, of which denoising contributes 720 ms — 88 percent. Every candidate optimization should be priced against that 720.

Decompose the step itself before optimizing it. Assume 3 ms of each 24 is CPU launch overhead — kernel launches and Python control flow rather than device math. Thirty steps spend 90 ms on launches, which graph replay can nearly erase if the served resolutions were captured at warmup; the BCG dive’s miss diagnostic is what tells you whether that assumption held in production. Caching then attacks the remaining ~630 ms of device work, parallelism its per-step critical path, and distillation the step count itself — three levers on the same 720, each with a different quality bill.

A cache that safely skips work equivalent to ten steps has a 240 ms upper-bound saving before lookup and correction — ten skipped steps at 24 ms each — and the accumulator design from the TeaCache dive is what decides which ten. Note the ceiling’s shape: even a perfect policy cannot touch the other 595 ms, so a deployment hoping for “2× faster” from caching alone needs a different arithmetic — fewer steps (distillation, better schedulers) or faster steps (parallelism, graphs), with caching as one term among several.

Doubling resolution changes latent work far more than text encoding. Doubling each spatial side quadruples latent tokens; attention over them can grow quadratically, so denoiser step time may rise several-fold while the 18 ms text stage barely moves. That is geometry, not measurement — the point is that the ratio between stages shifts with shape, so the bottleneck conclusion must be remeasured per served resolution. Separating stages helps only when reuse, independent scaling, or better batching repays intermediate transfer and queueing — the 6 MiB decoder boundary of the previous section is the price tag on one such split.

Practice: justify one optimization

Build a per-stage and per-step timeline from the numbers above. Test one caching policy at two resolutions and one parallel or disaggregated plan. Include graph buckets, synchronization, intermediate bytes, and every queue.

Report latency and throughput beside declared visual-quality metrics and blinded samples. State the workload boundary where your optimization loses. The worked analysis is in Appendix G.

The result should explain where time and bytes go, not announce that one optimization is universally best. Chapter 20 returns to language models in a different setting: inference embedded inside a training loop.

20. Inference for Reinforcement Learning

In online reinforcement learning for language models, inference does not serve an end user. It generates experience for a trainer.

The policy model produces one or more responses for each prompt. A reward or verifier scores them. The trainer updates the policy, and the new weights return to the inference workers for another round. The loop can repeat thousands of times, and every part of the serving system built so far in this book assumed the opposite of its defining property: weights that never change.

prompts -> rollout generation -> reward -> training update
   ^                                      |
   +----------- new policy weights -------+

This workload changes the engine’s lifecycle. Model weights are no longer immutable for the duration of the service. Caches derived from them, graphs compiled for them, and KV state produced by them all acquire version dependencies that a text service never had to express.

Rollouts arrive in groups and waves

Training algorithms often request several completions for the same prompt. Their shared prefix creates a strong cache opportunity — and it is the one kind of reuse this book has described that arrives pre-packaged: a group of 8 samples shares the entire prompt prefill, so the first sample’s prefill pays once and seven more ride the cached prefix. Chapter 16’s prefix machinery does the rest. Output lengths can be highly variable, especially for reasoning tasks. A group may not be ready for training until enough valid samples finish.

Online reinforcement learning couples serving and training through versions.

flowchart LR
    P["Prompts"] --> I["Inference rollout workers"]
    I --> R["Rewards and trajectories"]
    R --> T["Trainer"]
    T --> W["New policy weights"]
    W --> U["Versioned weight update"]
    U --> I

Rollout traffic also has a failure-domain property worth exploiting: a lost trajectory loses only its own compute, never a user’s request. That makes rollout workers good candidates for preemptible or bid-priced capacity — with one caveat. Losing a worker mid-rollout wastes its in-flight decode and its share of the group prefill unless trajectories checkpoint; whether spot capacity wins depends on reclaim frequency against the 171-second straggler arithmetic below, which is a measurable trade, not a slogan.

The scheduling consequence is brutal arithmetic. Take a group of 8 samples sharing one prompt: seven finish at 200 output tokens, one reasoning chain runs to 4,000. At the Atlas decode step of 45 ms, the straggler needs 3,800 × 45 ms ≈ 171 s more; for nearly three minutes, seven batch slots hold finished trajectories that training cannot use. The longest rollout holds up a synchronous batch while most workers become idle — not occasionally, but structurally, because reasoning-task length distributions are heavy-tailed by design.

Partial-rollout protocols exist for exactly this. The orchestrator can cancel, pause, or accept a subset according to the algorithm — some trainers learn fine from 7-of-8 groups, others require the complete set, and the serving layer must not decide on their behalf. The inference engine must report which policy version and sampling configuration produced every trajectory either way.

Scheduler fairness also changes. Advancing a nearly complete prompt group may unblock the trainer sooner than serving equal tokens across all groups — shortest-remaining-group-first, in effect. The right policy depends on the training algorithm’s data dependencies, which is why the queue’s ordering key should be group identity, not request arrival.

Bounding the loop with Little’s law

The rollout queue needs admission control like any queue in Part III, but the binding constraint is new: the trainer consumes batches at a rate the serving system does not control. Suppose rollouts complete one training batch every 60 s while the trainer needs 90 s per update. The mismatch accumulates — after three rounds, completed-but-untrained work represents two extra batches, and policy lag has grown by exactly that much because every waiting batch was generated under an older version. Appendix A’s Q = λW reads directly: steady queue size equals the completion surplus times the update period, so the fix is to throttle generation λ down to the trainer’s drain rate — pause admission of new groups once queued work exceeds roughly one batch, rather than letting finished trajectories pile up under ever-staler versions. The bound should be expressed in both bytes (memory) and versions (staleness), because either can be the binding constraint on a given round: a byte cap protects memory, a lag cap protects the algorithm’s assumptions, and G’s guidance is explicit that the serving layer must stop admission before trajectories grow unbounded while never inventing the staleness rule itself.

Colocate or disaggregate?

Training and inference can share GPUs in alternating phases. Colocation avoids dedicated idle pools and can transfer weights locally. It also requires careful memory handoff, and the arithmetic explains why. The Atlas policy’s weights are 140 GB; the trainer’s optimizer state at the usual 12 bytes per parameter of mixed-precision Adam (fp32 master, momentum, variance) adds roughly 840 GB across the cluster. No phase transition fits both live at once on the same silicon — colocated operation requires one side to vacate first, which is what sleep modes implement.

A disaggregated design gives training and rollout separate pools. Both can work concurrently, but weights must cross the network and rollout data can become stale — the policy keeps answering with version 41 while the trainer finishes version 42. Separation also frees each pool to be the right shape: the trainer wants dense compute with no KV machinery and can use different SKUs or reliability classes than the rollout fleet, and the rollout fleet can scale to the algorithm’s sample appetite without touching trainer state at all. The staleness budget, meanwhile, is a property of the algorithm — on-policy objective gradients tolerate little lag, while RLHF variants with KL anchors often train fine on mildly old policies — so the disaggregation decision is really a joint choice with the algorithm designer about which currency (bubble time, staleness, or network bytes) the project spends least on.

The AReaL paper studies a fully asynchronous system where rollout workers continue generating while training workers update the model. Its result is not permission to ignore staleness; the system and algorithm explicitly manage it.

The handoff timeline

Colocation’s cost is the bubble around every update, so price it stage by stage — declared assumptions throughout, since these numbers are deployment- specific. Pause admission, then let in-flight decodes drain: up to L_out steps at 45 ms, say 400 tokens ≈ 18 s for the unluckiest request. Sleep level 2 snapshots weights to host: moving 140 GB over a declared ~50 GB/s host link costs roughly 3 s (and the trainer cannot start until it completes). The trainer then runs — assume five minutes per round on the freed devices. Wake restores or replaces weights: if the new policy arrives by transfer rather than snapshot restore, 140 GB across NVLink-class links is ideal-case 140 ÷ 450 GB/s ≈ 0.3 s, with protocol overhead making single-digit seconds realistic. Invalidate KV and graphs, run one health forward pass, reopen. Total overhead lands near 30–60 s against a 300-s training step: a duty-cycle tax of ten to twenty percent, paid every round. That tax is exactly what the disaggregated design buys back at the price of network weight transfer and staleness management — there is no configuration without this bill somewhere, only different line items.

Sleeping is memory coordination

An inference engine can temporarily release or offload weights and KV state so a colocated trainer can use the device. Waking restores the required resources without rebuilding the whole process and distributed environment.

Different sleep levels may retain weights in host memory, discard KV state, or release both. The order matters during an update. Freeing KV memory before receiving new weights can reduce peak usage. The engine should not resume scheduling until weights and cache allocations are ready.

vLLM’s official sleep-mode documentation (release-dependent) describes releasing model and KV memory and selectively waking resources for RL workflows. In the pinned source the mechanism lives in gpu_worker.py, and it is worth reading as memory coordination rather than an allocator trick:

• sleep(level) takes a level: level 2 first snapshots every model buffer to host memory — {name: buffer.cpu().clone() for name, buffer in model.named_buffers()} — then calls suspend(level) on the sleep backend. Level 1 suspends without the CPU copy, trading wake fidelity for speed.
• The suspend is verified, not assumed: the worker records free_bytes_before_sleep, and after releasing, polls memory info until freed_bytes >= 0, asserting otherwise that “Memory usage increased after sleeping.” A sleep that silently leaked would hand the trainer a device that OOMs mid-step — the check converts that from a mystery into a failure at the right place.
• wake_up(tags) restores selectively: the "weights" tag replays the saved buffers back into the live model, the "kv_cache" tag calls post_kv_cache_wake_up(). A trainer that only needs the KV pool back can ask for it without paying weight restoration.
• The draft model gets its own snapshot set (_sleep_saved_draft_buffers) — a speculative-decoding deployment sleeps and wakes both models as one unit, which is the kind of coupled resource a naive “free the big tensor” view misses entirely.

SGLang exposes comparable sleep, wake, and weight-update controls through its engine and scheduler paths. The design point to carry forward is that sleep levels are an API surface between two systems with different memory owners: the trainer negotiates for capacity the way Chapter 6’s scheduler negotiates for KV blocks — explicit acquire, verified release, tagged restore.

The level choice itself prices cleanly with the handoff-timeline numbers. Level 1 wakes by reloading weights from storage — inside Chapter 17’s declared multi-minute cold-start envelope. Level 2 pays the ~3 s host snapshot up front and wakes by copying the same 140 GB back over the host link — seconds, not minutes. For an RL loop that sleeps and wakes every round, level 2’s snapshot is repaid the first time wake-up happens; level 1 exists for the rarer case where the device changes hands once and the snapshot traffic would compete with the very trainer it is making room for.

Weight update is a versioned transaction

A safe update has a beginning, a data phase, and a commit point.

A multi-rank update prepares everywhere before it commits anywhere.

flowchart LR
    S["Stage new weights"] --> V["Validate shapes and checksums"]
    V --> A{"All ranks prepared?"}
    A -->|No| R["Retain old active version"]
    A -->|Yes| C["Commit new generation"]
    C --> I["Invalidate dependent state"]
    I --> H["Health forward pass and reopen"]

1. Stop admitting model steps that would overlap the change.
2. Establish the transfer group and expected parameter metadata.
3. Move or refit every shard.
4. Verify completion across ranks.
5. invalidate state derived from old weights;
6. publish the new policy version and resume.

If one rank receives only part of the update, the model is not a slightly stale version—it is a corrupt mixture. The protocol must fail closed and recover all ranks to one version.

The new weights invalidate KV and encoder state produced by the old policy. They may also invalidate compiled artifacts if shapes or modules changed. Weight-only updates with an identical architecture can retain some graph structures, but the service should prove rather than assume compatibility.

Current vLLM documentation describes a four-phase pluggable weight-transfer protocol (release-dependent). SGLang’s updater lives in weight_updater.py with disk, distributed, tensor, and IPC entry points. The abstract steps map onto concrete entry points in both codebases:

The interesting difference is where each system spends its effort: vLLM makes the edges impossible to skip, while SGLang makes the silent failure modes impossible to enter. A production integration wants both properties, and neither codebase pretends the other half is unnecessary — the guards exist because sessions cannot check for cached weight splits, and sessions exist because guards cannot order a multi-chunk transfer.

Fail-closed in the pinned sources

Both codebases enforce the transaction’s edges, in complementary ways. vLLM’s worker wraps the data phase in an explicit session: update_weights refuses to run unless start_weight_update opened one (“start_weight_update must be called before update_weights”), and on any exception it deactivates the session and calls reset_weight_update_target() — the transfer aborts and forgets its destination, so a retry cannot append to a half-written model. finish_weight_update symmetrically refuses an unmatched call. On the trainer side, weight_transfer/base.py defines weights as a WeightSource with two channels: metadata() declares (name, wire dtype, full shape) “without transferring”, and iteration yields materialized pairs — where the docstring carries the participation rule one more time: “Materializing is typically a collective (FSDP full_tensor()), so every trainer rank must iterate the same source in the same order in lockstep, or ranks deadlock. Under pipeline parallelism a rank may not own a parameter at all — iterating still drives the collective and the yielded tensor is only meaningful on the sender.” Even reading the weight list is a collective.

SGLang’s contribution is the guard list — rejections for states where an in-place update would be silently wrong. The derived-weight check refuses online updates while a fused-GEMM optimization “caches the fp32 weight split; in-place loader writes are invisible to it, so an update would silently keep serving the old weights” — and the comment notes the check is “startup-determined and rank-uniform, so an update never proceeds on some workers while rejected on others,” rank-uniformity being what keeps the rejection itself from creating the mixed state it guards against. The IPC weight-cache check refuses while weights are shared with a daemon: “param.data is the daemon’s master copy shared with every co-attached engine, so an in-place update would silently corrupt them all.” And when a bucketed distributed update does fail mid-flight, the error text says what the transaction means: “The full weights of the ModelRunner are partially updated. Please discard the whole weights.” Not “retry the failed bucket” — discard, because a partially updated model is not a model.

On-policy does not mean one global pause

Strictly synchronous rollout keeps every sample tied to one policy version, but creates bubbles between generation and training. Fully asynchronous rollout keeps hardware busy and trains on older policies.

Several middle grounds exist, and they differ precisely in what staleness they introduce versus what idle time they tolerate:

The system can allow a bounded number of active policy versions or prioritize frontier groups that will complete the next training batch. Each row moves a cost between the bubble column and the staleness column; the algorithm decides which currency it can afford.

The algorithm determines which staleness is legal. The serving system must make version and group boundaries observable enough to enforce it — which is the coupled-resources table’s “version on every artifact” row turned into concrete signals: a policy-version stamp on every trajectory, a remaining-token gauge per group, and a lag histogram of trained-versus-generated versions. Frontier-first scheduling adds a subtlety worth naming: prioritizing groups by remaining tokens requires estimating remaining tokens, and the 171-second reasoning straggler from this chapter’s opening is exactly the case where the estimate is worst — the same heavy tails that create the bubble also blind the priority queue. Treat the estimate as another telemetry number with a staleness budget, not as ground truth.

Numerical agreement matters more in training

The trainer may recompute token log probabilities and compare them with values reported during rollout. Different kernels, precisions, templates, or batch shapes can create mismatches. Importance ratios can amplify small differences, and the amplification is multiplicative in a computable way: the per-token ratio is exp(logp_new − logp_old), so a logprob mismatch of ε appears once per token, and over a response of T tokens the sequence-level ratio carries the sum. Assume a BF16-scale logprob disagreement of 0.004 per token on a 1,000-token response: the sequence ratio is off by exp(4) ≈ 55× before any clipping. PPO-style clipping exists to survive exactly this, but a clipping mechanism that fires constantly is not training — it is discarding most gradients while appearing to run.

Record token IDs, masks, positions, model version, sampling state, and log probabilities with clear semantics. Decide whether the trainer uses inference engine values or recomputes them. The decision is a contract, not a preference: if the trainer recomputes, it must reproduce the inference engine’s kernel choices closely enough that ratios stay in range; if it consumes engine values, those values become part of the dataset format and must survive serialization exactly. Either way, the metadata recorded with every trajectory is what makes mismatches diagnosable later rather than mysterious forever.

Test long responses (where accumulated drift shows), padding boundaries (where masks decide whether pad positions contaminate sums), structured outputs (where grammar-constrained tokens may take different code paths), and MoE routing — Chapter 14’s expert-choice instability is a numerical-agreement hazard here too, since a different dispatch order changes the computed logprob of the same tokens.

Batch-invariant or deterministic modes help debugging, but may cost throughput. Use them to isolate differences even if the final production configuration is less strict — run one round in deterministic mode beside production, diff the logprobs, and you have a map of exactly which kernels disagree before it matters.

Worked example: prepare, then commit

Rollouts use policy version 41 while the trainer produces version 42 in inactive buffers. Every inference rank validates tensor shapes and checksums, then reports prepared(42). Only after all ranks prepare does the coordinator commit generation 42. Ranks swap buffers, invalidate version-dependent caches and graphs, run a health forward pass, and reopen admission.

Note what the validation step buys before anything moves: shape checks catch transposed or sharded-view tensors that would otherwise load silently into the wrong slots, and checksums catch truncated transfers that a byte-count comparison would accept. Both run against G’s manifest — “a manifest of tensors, shapes, dtypes, and checksums” — produced in staging storage, so validation is local to each rank and the coordinator sees only the boolean.

Walk the failure case, because it is the design’s whole point. Rank 3 fails mid-copy. No commit is published — the coordinator’s all-prepared gate never opens. Prepared ranks keep version 41 active and discard or retry their inactive buffers; the failed rank recovers into whichever state the protocol defines (re-transfer into its inactive buffer, or full rejoin). At no point did any tensor-parallel group contain mixed versions, because mixing is prevented at commit, not detected after: the active version changed for all ranks at one atomic instant or for none. This is a distributed transaction because a tensor-parallel group with mixed weights is not a valid model — not a slower one, not a slightly stale one, a nonexistent one.

The commit’s aftermath is the invalidation cascade from the colocated-memory discussion in reverse: KV entries produced under 41 are now wrong for scoring under 42, compiled graphs may survive (identical architecture) or may not (changed modules), and the health forward pass exists to prove the group actually generates before admission reopens. Only then do rollouts resume, stamped policy_version = 42.

Practice: fail one update safely

Trace rollout admission, reward, training, pause, staged transfer, prepare, commit, invalidation, health check, and wake. Attach the policy version to every trajectory, cache entry, graph, and message.

Fail rank 3 during transfer and show why no mixed group resumes. Then delay the trainer and define queue-byte and policy-lag admission bounds. Compare with Appendix G.

Inference inside training is still a serving system, but the customer is an algorithm with stronger version and reproducibility requirements. Chapter 21 examines another long-lived customer: the interactive session.

21. Interactive, Reasoning, and Agentic Systems

A voice assistant listens while the user speaks, begins responding, calls a tool, and stops mid-sentence when the user interrupts. There is no clean request followed by a clean response. Input and output overlap inside a long-lived session, the “prompt” is rewritten under the model’s feet as speech recognition revises itself, and the most important latency in the product is not time-to- first-token but time-to-silence after the user starts talking over the answer. The user steers constantly — interrupting, backchanneling (“mm-hm”), pausing, resuming — and the system must treat each of those as a scheduled event, not as noise to filter out.

Interactive inference is defined by deadlines, interruption, and suspended work, not merely low average latency. Voice and video systems expose stale output immediately. Reasoning models expose long, heavy-tailed generations; agentic systems add tool waits during which a request is alive but may not deserve accelerator memory. Every mechanism from Parts I through IV survives, but the scheduler must now understand more than a stream of decode tokens.

Build the latency budget backward

Suppose the product allows 700 milliseconds from the end of a user’s phrase to the beginning of audible speech. That budget includes network transport, speech recognition, endpoint detection, language-model queueing and prefill, first tokens, text-to-speech startup, and audio buffering.

A live conversation streams through several models and transports.

flowchart LR
    A["Audio input"] --> R["Streaming ASR"]
    R --> L["Language model"]
    V["Video or events"] --> E["Incremental encoder"]
    E --> L
    L --> T["Tools and retrieval"]
    T --> L
    L --> S["Streaming TTS"]
    S --> P["Playback buffer"]

Improving language TTFT from 250 to 200 milliseconds helps. It does not save a system that spends 600 milliseconds detecting the end of speech. This is why the budget is built backward — start from the user-visible interval, subtract stage by stage, and let the residuals name the priority — instead of forward, where each team optimizes its own component and the sum quietly misses.

Assign a budget to every stage and measure it at boundaries visible to the user, with one owner per stage. Audio generated but waiting in a playback buffer has not reached the user. A video frame completed after its display deadline may be useless even if throughput is high. Reserve an explicit margin rather than allocating to 100 percent — the stages’ tails are correlated (a CPU-spiking host delays ASR and TTS together), so a budget assembled from median stage times fails exactly when everything is slow at once.

Against this chapter’s declared stage costs, the allocation looks like:

The table’s discipline is that each row’s owner cannot spend another row’s budget: when the LLM stage wants 300 ms, the negotiation is explicit and the product owner sees which user-visible interval degrades.

Stream input as well as output

Ordinary generation receives a complete prompt. A real-time system can process partial audio, frames, or events as they arrive. An incremental encoder updates state without replaying the entire input.

The session may contain several coordinated models:

microphone -> speech recognition -> language model -> speech synthesis
                                      |
                                      +-> tools and retrieval
camera ----> vision encoder ----------+

Each arrow can stream. The streams also run on different clocks — audio at tens of chunks per second, camera events a few times per second, tool results arriving whenever the tool finishes — so a single global deadline is wrong; every stream carries its own age bound and the session reconciles them at the points where they meet. Buffers and queues need maximum ages, not only maximum sizes. Old audio or frames may be dropped because processing them would delay more current information.

Bidirectional transports such as WebSocket or streaming RPC carry events in both directions. The protocol should define ordering, heartbeats, flow control, reconnection, and which side owns the session. Reconnection is the hardest of these, and it has exactly two honest shapes: resume, where the client replays its last received sequence number per stream and the server continues from committed state (requiring the server to retain unacknowledged events, and requiring the session state that the next section discusses to have an owner); or restart, where the session re-initializes visibly — new turn generation, fresh endpointing, and a transcript rebuilt from history. A protocol that silently mixes the two produces duplicated or missing audio with no way to diagnose which happened. Transport keepalive is not the same as model-worker health — a socket can be open while the engine behind it has wedged, which is why Chapter 17’s health semantics apply per component, not just per connection.

Queues bound age, not size

The reason age bounds replace size bounds deserves the arithmetic. Media consumes at real-time by definition: one second of audio takes one second to play. Suppose a transient stall leaves 500 ms of audio queued. If the system diligently plays everything, the backlog drains at exactly real-time — every subsequent sample, including everything recorded after the stall, is now delayed by 500 ms forever. No amount of later throughput fixes it, because the pipe’s width is fixed by the clock, not by capacity. The only exit is dropping: when a chunk’s age exceeds its deadline, discard it and resync to live — trading a gap the user hears once for a lag the user hears forever. This inverts the reliability instinct built into every other queue in this book. TCP-style guaranteed delivery is precisely wrong for live media; the right contract is bounded-staleness delivery, where the queue’s job is to decide what to skip, not what to hold. The same logic sizes the playback buffer: deep enough to hide jitter, shallow enough that an interrupt command issued now reaches silence quickly — buffer depth is interruption latency waiting to happen.

Speech is a chain of clocks

Automatic speech recognition is not merely a file-to-text model placed before an LLM. A streaming recognizer emits partial hypotheses that may be revised as more audio arrives. Endpoint detection decides when a phrase is complete. If the endpoint waits too long, the language model starts late; if it fires too early, the prompt is incomplete.

Partial hypotheses create a hidden coupling back into Part III: every token the LLM prefills against a stale hypothesis is work that a revision invalidates. If the recognizer revises “book a table for two” into “book a table for twelve,” the KV blocks computed past the revision point are wrong — Chapter 16’s prefix reuse, running in miniature, once per hypothesis update. A well-built pipeline therefore gates LLM starts on hypothesis stability (or prefills only the stable prefix), accepting some start latency to avoid paying repeated invalidated prefills.

Google’s work on joint endpointing and decoding frames this as a quality-latency trade rather than a preprocessing detail.

Record three ASR times separately: audio arrival to first partial text, end of speech to stable transcript, and the age of audio at every emitted hypothesis. The last one catches a system that produces frequent updates from an ever-growing backlog. The distinction is visible in G’s timeline: the partial arriving at 0.8 s describing speech spoken since 0.6 s has an audio age of about 200 ms — live, useful. A partial arriving at 3.0 s that still describes audio from 1.0 s has an age of two seconds: the recognizer looks productive (three updates!) while actually narrating the past. Downstream stages that trust recency will act on stale text; the age metric is what makes the backlog visible before that happens.

Text-to-speech has an equally important split. Time to first audio measures startup, while real-time factor compares synthesis duration with audio duration:

real-time factor = synthesis time / generated audio duration

A factor below one is necessary for sustained streaming, but not sufficient for a good conversation. The first chunk can still arrive late, chunk boundaries can click, or the playback buffer can grow until speech no longer matches the current turn. Measure first-audio latency, chunk cadence, underruns, buffer depth, and interruption-to-silence.

Suppose ASR stabilization takes 180 ms, LLM first token takes 220 ms, TTS first audio takes 140 ms, and network plus buffering uses 90 ms. The total is 630 ms. A 20 percent faster decoder saves 44 ms, while better endpointing that saves 100 ms has more than twice the product impact. Stage budgets turn an attractive kernel result into the correct system priority — and note where the 180 ms of endpointing sits: it is waiting, not compute, which is why it yields to better decisions rather than better hardware.

End-to-end streaming models change the component boundaries but not the need for explicit latency and quality measures. Meta’s SeamlessStreaming research is one example of evaluating streaming speech generation with latency, robustness, and perceptual criteria together.

Interruption is a first-class transition

When the user begins speaking over the assistant, the service may stop audio playback, cancel text-to-speech, stop future language scheduling, and decide how much of the generated but unheard text belongs in conversation history. Even detecting the interruption is a pipeline decision: a voice-activity detector watching the input while TTS plays must distinguish a real barge-in from the assistant’s own voice arriving through the microphone — echo cancellation does most of the work on device, and where it cannot, deployments fall back to half-duplex politeness or require the new speech to exceed an energy threshold for some duration. Every one of those mitigations adds detection latency to the interruption-to-silence budget before any cancellation even starts.

Interruption advances the session generation and fences late work.

flowchart LR
    G7["Turn generation 7"] --> O["Text and audio in flight"]
    I["User interruption"] --> G8["Advance to generation 8"]
    G8 --> C["Cancel scheduling, synthesis, and playback"]
    O --> X{"Event generation current?"}
    X -->|No| D["Discard late event"]
    X -->|Yes| P["Deliver event"]

Those actions cannot happen atomically across several services. Use a session generation number or turn ID. Events from an old generation are ignored after the interruption advances the session.

Cancellation should flow upstream quickly, but components must also clean up late completions. A tool call may finish after the turn has been abandoned. Its result should not silently enter a new turn.

Fencing with generations

The generation counter is the chapter’s central data structure, and it is worth spelling out the contract each component signs. On advancing from generation 7 to 8, the session controller bumps the number first; every other action follows from components observing it. Language-model scheduling checks the generation before admitting the next step’s work. The TTS service checks before synthesizing each chunk. Playback checks before rendering each buffer. Tool results carry the generation they were requested under, and the history-commit rule consults it: only text actually heard — delivered to playback under the then-current generation — enters the visible conversation; generated-but-unheard text is recorded as diagnostic state, never silently treated as spoken.

Readers have met this pattern twice before. It is Chapter 17’s membership epoch (a stale health report self-invalidates) and Chapter 20’s policy version (a trajectory stamped with an old version cannot contaminate a new round). Real-time interruption adds only one twist: the fence must be checked at consumption time, not just dispatch time, because the consumer — the user’s ear — is the one component that cannot be rolled back. Five hundred milliseconds of stale audio played after an interrupt is a worse failure than five hundred milliseconds of compute wasted generating it, which is why the interruption-to-silence budget in G’s worked timeline (100 ms) is the tightest number in the chapter.

The cancellation order follows from each consumer’s granularity. Bumping the generation is microseconds and must go first — it is the only step that makes every later race benign. Stopping playback is next and is bounded by buffer depth, which is why the age-vs-size dive called depth “interruption latency waiting to happen.” Cancelling TTS takes effect at chunk boundaries; the in-flight chunk is wasted but fenced. Cancelling language-model scheduling takes effect at engine-step boundaries — at the Atlas decode cadence, up to one step’s worth of batch compute continues before the check runs, which is the worst-case wasted compute and is bounded and acceptable. Tool calls are the long pole: they may run for seconds, cannot be revoked, and are handled purely by fencing at completion — their results become diagnostics if the generation moved. The pattern: cancel where granularity is fine, fence where it is not, and always bump the fence first.

Session state needs an owner

A live session can own audio buffers, encoder state, conversation tokens, KV blocks, parser state, tool calls, and output already sent but not yet played.

Keeping all state on one worker simplifies consistency and creates affinity. Worker loss then ends or reconstructs the session. Externalizing selected state supports migration at the cost of serialization and latency.

Classify state by recovery value. Conversation text is compact and easy to store. KV state is larger but saves prefill — at the Atlas constant of 320 KiB per token, ten minutes of dense conversation accumulates KV that dwarfs its own transcript by orders of magnitude, which is why externalizing KV wholesale is rarely worth it and checkpointing prefix boundaries often is. A causal video model’s recurrent state may be both large and expensive to reconstruct. Checkpoint frequency should follow the product’s recovery requirement: how many seconds of conversation is a worker loss allowed to cost?

Migration, when chosen, inherits Chapter 17’s sticky-session escape rules — affinity is a preference evaluated per turn, never a requirement the user’s latency pays for.

Checkpoint frequency has its own arithmetic. If the product tolerates losing R seconds of conversation, checkpointing roughly every R/2 bounds the expected loss near R/4 to R/2 while amortizing the write. The cost per checkpoint is state size over write bandwidth — trivial for the transcript’s kilobytes, material for recurrent media state — so the checkpoint cadence should differ per state class: conversation text at turn boundaries, KV prefix anchors at stable endpoints only, media state only if the recovery contract demands it. Checkpointing everything at one cadence either wastes bandwidth on cheap state or under-protects expensive state. In-flight tool calls and parser state need an owner decision too: if a worker dies holding a pending tool invocation, the recovering session must either re-issue it — needing a stable request ID so the tool can suppress duplicates, per Chapter 17 — or leave it abandoned and let the turn’s fence mark its eventual completion as diagnostic. Silence in the meantime must be bounded by a timeout, because a hung tool otherwise becomes a session that never responds again.

Reasoning changes the workload shape

A reasoning model may emit an internal reasoning stream before its visible answer, use a parser to separate the two, and vary that work by orders of magnitude across superficially similar prompts. Admission therefore needs a budget for total generated work, not merely visible answer length. Traces must record reasoning and answer tokens separately so that a product cannot appear to reduce latency by hiding work from its accounting.

An agentic turn alternates active generation with suspended work.

flowchart LR
    U["User turn"] --> R["Reasoning generation"]
    R --> P["Reasoning and tool parser"]
    P -->|Need evidence| T["Tool call"]
    T --> W["Suspended request"]
    W -->|Result and valid fence| R
    P -->|Ready| A["Visible answer"]

Chapter 22 owns the parser and wire contract. This chapter owns what that contract does to scheduling: reasoning tokens consume decode slots, parser state must survive interruption, and a disabled or shortened reasoning mode is a quality-tier decision rather than a free speed switch. The current SGLang reasoning-parser documentation shows the explicit separation, while the vLLM feature index tracks the serving controls that interact with it.

Compressing or discarding reasoning history can reduce the next turn’s prefill and KV footprint, but it changes model input and may change answer quality. Treat the summary algorithm and version as cache identity, evaluate the quality trade, and apply Chapter 26’s retention rules to both the original reasoning and its summary.

Tool gaps create suspended requests

While a tool runs, retaining all KV blocks buys a fast resume and charges scarce memory to idle wall time. Releasing them saves capacity and later pays a prefill or restore cost. Make that choice from expected tool latency, state size, available cache tiers, and the turn’s remaining deadline. A short database read may justify retention; a human-approval step usually does not.

Suspension needs a stable request identity, an expiry time, and a generation fence. When the tool returns, the router must find the session owner or restore its state, then verify that the turn is still current before resuming. Tool execution also needs its own idempotency key: retrying a generation is safe only when it cannot repeat an external action. Late results become diagnostic data, not new model input.

Graceful degradation is a scheduler policy

During overload, a real-time service cannot let queues grow indefinitely. Possible responses include lowering video resolution or frame rate, using a smaller model, shortening speculative lookahead, skipping optional tools, reducing generated speech, or rejecting a new session.

Choose degradation in product terms, and order the ladder before overload arrives. Dropping every other frame may preserve a conversation; allowing two seconds of stale audio may destroy it — perceptually, staleness is worse than poverty. A defensible ladder spends cheap quality first:

Each tier is a scheduler policy with its own trigger metric — queue age, deadline-miss rate, session count — not an operator’s manual intervention. Tier 1 refunds slack immediately, as Chapter 19’s sessions do; tier 5 is last because it converts overload into unavailability. The scheduler needs deadlines and quality tiers to execute any of this mechanically rather than as ad-hoc throttles.

Fairness also changes. A long-lived session should not own a permanent batch slot while silent. An active speaker may deserve temporary priority. Use quotas so one tenant’s continuous sessions cannot exclude new users.

Observe a conversation, not isolated calls

Per-request TTFT does not capture a voice conversation. Measure end-of-turn to first audio, interruption-to-silence, gap and overlap duration, stale event rate, tool delay, session recovery, and perceptual quality. Gap and overlap deserve definition because they come from the same annotation and point in opposite directions: gap is silence between the user finishing and the assistant starting (or between turns) — awkwardness the user feels as hesitation; overlap is speech during assistant audio — either barge-in (fine, the interruption machinery exists for it) or the assistant still talking over a finished user turn (never fine). Overlap rate that survives the fence is a direct quality regression; gap percentiles are where “it feels slow” complaints live even when every stage met its budget.

Trace events with session ID, turn generation, component timestamp, and clock source. Distributed clocks are imperfect, so include monotonic stage durations and propagate trace context. Session recovery is itself a measured event with two distinct shapes — resume onto a warm worker versus restart from history — and users experience them differently, so track recovery rate and recovery latency per shape rather than blended. The stale-event rate deserves emphasis because it is a leading indicator: events discarded by the generation fence are the system reporting, quantitatively, how often cancellation lost the race — a rate that climbs before users start complaining about assistants that talk over them.

Worked example: a late audio chunk

One coherent reading of G’s timeline: the user stops speaking at 2.4 seconds; partials had arrived at 0.8, 1.5, and 2.2. Endpointing stabilizes at 2.6, the LLM routes and prefills by 2.8, emits a tool call at 3.0, a holding phrase covers the tool wait, and response audio begins at 3.8 — an end-of-turn-to- first-audio span of 1.4 s, decomposed in G as 200 ms endpointing, 200 ms language startup, 600 ms tool time overlapped with the holding phrase, and 200 ms TTS plus buffer. The holding phrase is itself a mechanism worth naming: a short spoken response (“One moment —”) synthesized immediately so the user hears acknowledgment while the tool runs, at the cost of a second, tiny TTS job racing the real one through the same synthesis pool. At 5.1 the user interrupts, advancing the turn generation from 7 to 8; playback becomes silent by 5.18 — the 80 ms inside the 100 ms interruption budget. A generation-7 audio chunk arrives at 5.4 and is discarded by the fence, exactly as designed.

Two budgets governed different halves of those ten seconds, and they fail differently. Missing the 1.4-second response budget feels slow; missing the 100-ms silence budget feels broken — the assistant talking over the user is the canonical real-time failure. The counterfactual shows why the fence earns its complexity: without it, the chunk arriving at 5.4 plays after silence was restored at 5.18, exposing the user to 220 milliseconds or more of stale speech resuming mid-word — precisely the failure users describe as “it wouldn’t stop talking.” The generation number makes distributed cancellation coherent across models that cannot share a lock; every event additionally carries a session ID, stream sequence number, and deadline, so ordering, duplication, and staleness are all independently detectable. And the discard at 5.4 is the system working: generated-but-unheard text was never committed as something the user heard.

Practice: specify a ten-second protocol trace

Draw partial ASR, endpointing, LLM prefill, tool execution, TTS, playback, and interruption on one timeline. Budget end-of-turn to first audio and interruption to silence separately. Delay the tool and deliver a stale audio event after cancellation.

Write ordering, generation, backpressure, history-commit, and cleanup rules. The complete worked timeline is in Appendix G.

Parts I through IV have focused on execution mechanisms. Part V turns them into a production contract: APIs, experiments, operations, economics, and security.

Part V — Production Discipline

Mechanisms become a service only through the discipline around them: an API that holds still while the engine changes, benchmarks that can be believed, a debugging method that turns symptoms into fixes, operations that can diagnose themselves at 03:00, architecture decisions that name their own review triggers, and security boundaries that follow data and authority.

Chapters 22–26, followed by the production debugging playbook in Appendix I

22. APIs, Streaming, and Structured Generation

An engine can generate the right token and still return the wrong response.

Perhaps a chat template inserted the wrong role marker, so a fine-tuned model receives a conversation shape it never saw during training and answers as something else. A streamed tool call arrived out of order, so a client executed arguments meant for a different call. Usage counted cached input twice, doubling someone’s bill. A network retry produced two side effects — two charges, two emails — because the second attempt was not distinguished from the first. These failures live above the model, but they are part of inference correctness: users experience them as the system being wrong, not as an API layer having opinions.

The API is the boundary where a changing engine promises stable behavior to its callers. Everything behind it may be rewritten every week; the boundary may not move without a version.

An endpoint is more than a URL

A production server may expose chat, text completion, embedding, scoring, classification, reranking, image, audio, or real-time endpoints. Each endpoint defines accepted inputs, defaults, limits, output fields, streaming events, and error behavior — and each of those definitions is load-bearing. A default maximum output length changes what “the model said” means; a silent truncation changes it again.

The API translates a public contract into engine work and back again.

flowchart LR
    C["Client request"] --> A["Authenticate and authorize"]
    A --> V["Validate limits and semantics"]
    V --> T["Template and tokenize"]
    T --> E["Engine request"]
    E --> O["Output events"]
    O --> P["Protocol framing and usage"]
    P --> C

Protocol compatibility should be stated at that level. Two servers can both accept a familiar chat request while differing on unsupported parameters (ignored? rejected? echoed?), tool-call deltas, usage events, whether stop-token text appears in the returned content, or error codes. Test the semantics your client uses instead of relying on a compatibility label — Chapter 23’s benchmark discipline applies to conformance just as much as to speed.

Version behavior that affects output. A new parser or chat template can change responses as materially as new weights, which is why Chapter 18 treated processor version as execution identity and this chapter treats template and parser revisions as part of the served artifact’s name.

Errors deserve the same specification effort as successes. A useful taxonomy separates at least four classes with distinct client obligations: malformed request (fix the request), limit exceeded (shed load or raise quota — see the limit pricing below), capacity unavailable (retry with backoff, ideally elsewhere per Chapter 17’s routing), and internal failure (retry only if idempotent). Collapsing them into one 500-with-a-string forces every client to reimplement the classification, badly and independently.

Tokenization is part of the contract

Chat messages must become the token sequence expected by the model. The chat template controls role markers, separators, tool definitions, and generation prompts. Tokenizer files define how text and special tokens map to IDs.

If a client sends token IDs directly, the service needs a rule for whether it trusts them, adds special tokens, or verifies context length — each choice produces a different model input from the same bytes, and only one of them is what the caller intended. If the server returns log probabilities, callers need to know which tokenization produced them; a logprob over your retoken- ization of my text is a number attached to nothing.

Record the tokenizer, template, and processor revisions with the model. A weight-only model name does not fully identify the served behavior — two deployments of identical weights can disagree on every response through the template alone.

Stop conditions are contract too, and they hide a tokenization decision. A stop string is matched against decoded text, so its meaning changes when the tokenizer merges or splits characters differently across revisions; a stop token ID pins the exact vocabulary entry but stops matching if the template rewrites the surrounding text. The API should say which form is authoritative, whether matched stop text is included in or excluded from the returned content, and how max_tokens truncation reports itself distinctly from a clean stop — Chapter 23’s benchmark harness depends on that distinction, since a length-truncated run is not a completed sample.

Streaming is a state machine

A stream is not a series of unrelated JSON objects. It has a beginning, ordered deltas, optional tool or reasoning channels, a finish reason, usage, and a terminal event.

A streamed request remains a state machine after the connection changes.

flowchart LR
    Q["Queued"] --> R["Running"]
    R --> S["Streaming"]
    S --> F["Finished"]
    Q --> C["Cancelling"]
    R --> C
    S --> C
    C --> D["Device work drained"]
    D --> X["State released"]

The server should define whether usage appears only at the end, whether a tool call’s name can stream separately from its arguments, and how partial UTF-8 or token boundaries are handled — a multi-byte character split across chunks must reassemble identically regardless of where the network cut. Clients should tolerate network fragmentation without reordering semantic events: fragmentation changes chunking, never meaning.

Once a terminal event is emitted, no later content belongs to that attempt. If the worker finishes after the client disconnects, the server still needs to release request and parser state — the connection ending does not end the state machine, which is why the second diagram runs past cancelling to drained device work and released state.

Backpressure belongs inside this machine rather than beside it:

flowchart LR
    T["Engine output tokens"] --> B{"Connection buffer<br/>below limit?"}
    B -->|Yes| E["Emit event downstream"]
    B -->|No| H["Hold: stop draining<br/>this request's output"]
    H --> R{"Client resumed reading?"}
    R -->|Yes| E
    R -->|"No, deadline exceeded"| X["Treat as disconnect: cancel"]

A slow consumer holds only itself. The hold must never block the shared path from engine to other connections — G’s worked rule — and a held stream that never resumes is a leak with a deadline, so the hold converts to cancellation when its deadline expires.

Terminal events and late arrivals

Event order is part of the machine. A conventional ordering: content and tool deltas while running, then finish_reason, then usage, then nothing. Each clause excludes something specific. Usage-last lets a client bill only from terminal frames without scanning deltas. Finish-reason-before-usage lets a client decide whether the usage matters (a length-truncated answer bills differently than a complete one). Nothing-after-terminal means a client can treat the terminal event as permission to free its buffers.

The last clause needs enforcement, not convention. Suppose the client disconnects at t and the worker finishes its in-flight model step at t + 40 ms, emitting the final chunk to a connection that no longer exists. The correct behavior is to drop the chunk at the connection writer, and the mechanism is the generation fence from Chapter 21, narrowed to one request: every output event carries its attempt’s generation; the terminal event closes that generation; any later event bearing it is stale and discarded. Without the fence, a client that retried after the disconnect receives interleaved fragments of two attempts. The r-17 conflict rule prevents two live writers; the fence handles the zombie writer that outlives its connection.

Mid-stream failures need a decision, because different failures deserve different shapes:

The second row is easy to get wrong. A grammar that terminates early did not fail — refusing to emit invalid JSON is the feature working. Collapsing it into an exception-shaped close teaches clients to retry requests whose constraint genuinely admits no continuation, burning capacity on a request that can only fail again.

Structured output moves validation into decode

JSON Schema, regular expressions, and grammars restrict which tokens are legal at each step. They can prevent malformed output instead of validating and retrying after generation — trading one guaranteed pass for a speculative pass plus repair, usually a good trade at Chapter 5’s prices.

The API must distinguish a schema that cannot be compiled from a generation that reaches no valid continuation. It should report which subset of a standard is supported. Grammar compilation can be cached, but the cache key must include the backend and grammar version — two backends compile the same JSON Schema into different token machines, and a cache hit across them silently serves one product’s grammar under another’s name.

Grammar state advances beside model state on every sampled token.

flowchart LR
    L["Model logits"] --> M["Grammar or schema mask"]
    P["Parser state"] --> M
    M --> S["Sample legal token"]
    S --> U["Update parser state"]
    U --> P
    S --> O["Ordered output stream"]

A 128k-token vocabulary needs 128k legality bits per position: 16 KiB as a packed mask. At batch 64 with three speculative positions plus the bonus token, four rows per request require about 4 MiB of masks per engine step. That is small beside model traffic but not free, especially because host-side parser state must produce it before sampling. Schema complexity therefore belongs in both admission limits and performance measurements.

Tool and reasoning parsers interpret model-specific token conventions. They are streaming parsers because a complete object may arrive over many tokens. Parser state belongs to one request and must be updated in the same order as output tokens.

At the pinned snapshots, vLLM’s protocol and parser implementations span entrypoints/openai and vllm/parser — the latter holding per-model tool and reasoning parsers plus their metrics. SGLang implements compatible endpoints and parser paths under srt/entrypoints/openai and its constrained-decoding package.

These are fast-moving interfaces. Pin behavior with protocol tests.

Constraints at speculative positions

Speculation gives every proposed position a provisional parser state. A token forbidden at position two cannot be accepted merely because positions zero and one were legal; verification commits both model and grammar state only through the accepted prefix and rolls both back at the first rejection.

At the pinned vLLM snapshot, StructuredOutputManager sizes its bitmask tensor for max_batch_size * (1 + max_num_spec_tokens) rows—one legality row per speculative position plus the bonus token. The implementation lives under vllm/v1/structured_output. The design rule is framework-independent: ordinary and speculative decoding must consult the same constraint machine, and a benchmark must use the schemas that production traffic actually carries.

Which constraint, and what happens when it cannot exist

Choosing a grammar backend is itself API behavior, and SGLang’s selection path in create_grammar_backend encodes a posture worth copying. Selection order: a registered custom backend wins; otherwise a named backend — xgrammar, outlines, llguidance, or none. Then the interesting part: if XGrammar cannot initialize for this tokenizer (TokenizerNotSupportedError), behavior splits on a flag. With enable_strict_thinking, it raises — the error says strict thinking “requires a grammar backend with token filtering support” and “Cannot fall back to grammar_backend='none'”. Without it, the server logs a warning and falls back to none, where structured outputs “will not be available.”

The same missing capability produces two different outcomes, and both are correct, because the question is whether any caller’s contract depends on the constraint. Strict thinking means token filtering applies inside reasoning spans; silently dropping it changes emitted behavior for callers who opted in, so it fails closed at startup — the same philosophy as Chapter 20’s guard list, where an unsafe combination refuses to boot rather than misbehaving later. Optional schema support degrades with a log line instead, because no request promised it. The API-layer translation: report which constraints a deployment enforces in its self-description, and refuse to start when a load-bearing one is absent.

One more piece of machinery earns mention. Batched mask fills want a preallocated vocab-sized mask tensor, and register_vocab_mask_buffer validates any re-registration against the existing buffer’s shape, dtype, and device — a mismatch raises rather than quietly swapping the tensor every sampler reads from. Like Chapter 20’s weight-cache guards, it is a startup-determined, rank-uniform check: either every rank agrees on the buffer or the process fails loudly, never a mixture.

Inside a grammar backend

SGLang’s constrained-decoding base class — base_grammar_backend.py at the pinned SHA — shows how much machinery hides under “restrict which tokens are legal.” The per-request object is a BaseGrammarObject whose lifecycle is three calls the sampler makes: accept_token(token) after each emitted token advances the machine, rollback(k) rewinds it when verification rejects a suffix — the same rollback Chapter 11’s speculative decoding forces on the model state, here applied to the grammar state in lockstep — and is_terminated() when no legal continuation remains. Mask mechanics are deliberately batch-aware: fill_vocab_mask_batched fills “listed rows, leaving unlisted rows untouched,” so constrained and unconstrained requests coexist in one step’s mask tensor, and a GrammarMask carries “any one of the batch’s” grammars as “a handle, not per-request state.”

Compilation never blocks the way a naive implementation would. get_cached_or_future_value checks a (type, string)-keyed cache and returns a per-request copy() on hit; on miss it submits compilation to a thread pool and returns a Future — the request proceeds toward decode while the schema compiles elsewhere, waiting only where the mask must exist. Every compiled object carries GrammarStats: compilation_time, ebnf_size, is_cache_hit, even num_timeout and is_grammar_aborted, which is the observability needed to answer “did schemas make this slower?” without a profiler. And the compile-failure distinction this chapter demands exists as a type: an uncompilable schema becomes InvalidGrammarObject, “carrying the original error message,” while unsupported subsets degrade through _not_supported with a logged skip — the API surfaces why there is no grammar instead of inventing a dead-end mid-generation.

Cancellation, deadlines, and retries

A client deadline should propagate through the router and engine. Work that cannot produce a useful result before the deadline should stop consuming capacity — Chapter 17’s admission veto applied continuously rather than once. The server may distinguish client cancellation from its own overload or internal timeout because callers respond differently: a cancelled request should not be retried by infrastructure, an overloaded one maybe should.

Where the deadline actually binds

Walk a first-byte deadline of 300 ms through Atlas’s frozen cost model. At admission the predicted path is: 150 ms queued (declared assumption for this walk), then prefill_ms(2000) = 20 + 70 = 90 ms, then one decode step of 45 ms before the first byte — 285 ms, under the deadline by 15 ms. Now the queue runs long and the same request faces 180 ms of waiting: first byte at 315 ms. The prediction error lives entirely in the queue term; prefill and decode costs are stable, which is why Chapter 17 scored placements as queue plus known costs.

Two designs handle the drift. Admit-anyway-then-cancel: the server accepts, starts prefill at 180 ms, and the client gives up at 300 — the system burned 120 ms of prefill plus up to one 45 ms step of finished work and delivered nothing. Continuous veto: when the predicted first byte exceeds the remaining budget, the request is rejected before prefill begins and never occupies KV. The veto converts a guaranteed disappointment into an immediate, cheap, actionable error — and because the queue term dominates the error, the veto should be re-evaluated whenever the queue estimate moves, not only once at admission.

Cancellation granularity follows from engine structure rather than API preference. A cancel arriving mid-model-step cannot un-commit the step; the bound is one engine step of wasted work (Chapter 5’s step structure, the same bound Chapter 21 accepted for interruption). Promising “instant” cancellation in the API would be promising something the executor cannot deliver — the contract should say “stops within one engine step,” which is testable.

Retries are safe only when the operation is idempotent or carries a stable request ID. Text generation without side effects can usually be attempted again, although the sampled output may differ. A tool-executing endpoint may have already charged a card or sent a message.

Separate generation from external action. Give tool executions their own idempotency keys and authorization checks. Never treat model output as trusted instructions merely because it matches a schema — validity is syntax, and Chapter 26’s security discussion owns what validity does not cover.

The duplicate-ID contract deserves its exact terms, since it is the kind of clause teams discover they need only after double-charging a customer. For the worked example’s r-17: Atlas rejects a second live attempt with a conflict status — two engines running one request ID simultaneously is always a bug, never a retry strategy. After completion, an idempotent request may return its recorded terminal result for a retention window, which converts a transport retry after success from duplicate work into a cache read. Tool execution uses a separate idempotency key because regenerating text and repeating an external action are not equivalent operations: the first is safe to redo, the second is the whole reason the contract exists.

Authentication and resource limits

Authentication identifies the caller. Authorization decides which models, adapters, tools, and data it may use. Quotas and rate limits protect shared capacity.

Token limits alone are insufficient for multimodal inputs or expensive sampling modes — a 128-token limit means nothing to a 40,000-patch image. Limit decoded media, number of candidates, grammar complexity, tool definitions, output length, and concurrent sessions. Apply limits before allocating model state where possible: rejecting an oversized request before it owns KV blocks costs nothing, evicting it after costs everyone.

Pricing a limit check

The claim “reject early” has an arithmetic spine. Atlas KV state costs 320 KiB per token (Appendix A). A request with an 8,000-token input allowed 4,096 output tokens owns up to about 12096 × 320 KiB ≈ 3.7 GiB of KV at peak. On a card whose weights consume 35 GiB under tensor parallelism of four (140 GB sharded) plus activation headroom, suppose roughly 30 GiB remains for KV: eight peak-size sequences fill it. A per-key concurrency limit of eight is not bureaucracy; it is the difference between serving eight well-shaped streams and letting the ninth trigger eviction machinery that degrades all streams — Chapter 6’s preemption, arriving through an API misuse.

And the two failure orders price differently. Rejecting before allocation costs microseconds of validation. Admitting then evicting burns the victim’s prefill — at 0.035 ms per token, an 8,000-token prefill is 280 ms of finished GPU work discarded — and taxes whoever shared its batch with the eviction step, and still owes the caller an answer. Limits checked at the boundary are cheap precisely because they run where nothing expensive exists yet.

This is also why limits belong in error semantics, not just enforcement: the rejection should say which limit fired (concurrency, context_length, media_bytes) so clients can shed load intelligently instead of retrying a request that will never fit.

Avoid exposing administrative engine APIs—weight updates, sleep, arbitrary collective calls, cache control, or profiling—on the public inference network. They can change model behavior or deny service. Chapter 20 made weight updates transactions; making them internet-reachable undoes that care with one routing mistake. Separate credentials and network boundaries are appropriate.

Worked example: disconnect is a state transition

A slow client fills its bounded output buffer. Per the backpressure diagram, the server pauses that stream’s output drain without blocking the shared output path; other requests continue streaming. When the connection closes, the request enters cancelling; future model steps stop, in-flight output is ignored, and KV references eventually return to the baseline count. Each clause is testable: steps-stop has a bound of one engine step, ignored-output is the generation fence doing its job, and the KV count returning to baseline is the assertion that no reference leaked.

Give the pause a timeline with declared numbers. The buffer fills at t = 0 and the drain holds; the engine, mid-step, finishes the current model step at t = 45 ms and does not schedule further output for this request. At t = 120 ms the client’s socket dies. The request moves to cancelling; its generation counter bumps so any event emitted by the finishing worker is stale at the writer. At t = 165 ms the last in-flight step completes and the request’s KV references release — total wasted work from disconnect onward: one engine step, 45 ms, exactly the promised bound. A conformance test asserts each transition: buffer-full must pause within one poll interval, cancel must stop scheduling by the next engine step, and KV must return to baseline within a bounded drain window rather than “eventually.”

Duplicate request ID r-17 also needs a contract. Atlas rejects a second live attempt with conflict status; a completed idempotent request may return its recorded terminal result for a retention window; tool execution uses a separate idempotency key because repeating generation and repeating an external action are different operations. G’s conformance framing then closes the loop: run the suite against old and new engine revisions and classify every difference as intended API change, allowed numerical variation, or regression — “both returned HTTP 200” is not conformance.

Practice: build a semantic conformance suite

Pin a tokenizer and chat template, then test streamed and non-streamed results, stop conditions, log probabilities, usage, schemas, tool calls, errors, and cancellation. Add a slow consumer, mid-generation disconnect, duplicate live ID, and retry after completion.

Assert semantic output, event ordering, bounded backpressure, and eventual GPU state release. Classify engine-version differences instead of checking only HTTP status. The worked contract is in Appendix G.

23. Benchmarking and Performance Science

A benchmark is an experiment designed to answer a decision. Without the decision, it becomes a number generator.

“Which engine is fastest?” is too broad. A useful question is narrower:

For our model, hardware, request distribution, and TTFT/ITL targets, does prefix-aware routing increase goodput enough to justify its control-plane complexity?

That question tells you what to hold constant, what to vary, and which result matters. It also tells you what the benchmark is allowed to cost: a question that gates a routing rewrite justifies a week of careful measurement; a question that gates a kernel flag does not. Budget the experiment like any other engineering artifact.

The failure mode this chapter exists to prevent is the confident wrong number: a result that is precisely measured, cleanly plotted, and answers a question nobody asked — or worse, answers it under conditions so different from production that the decision it drives is a coin flip with extra steps.

Begin with a hypothesis

Write the expected causal chain before running the test. For example:

A benchmark is an evidence loop, not a single load-generator run.

flowchart LR
    H["Falsifiable hypothesis"] --> W["Representative workload"]
    W --> R["Controlled repeated runs"]
    R --> A["Raw events and analysis"]
    A --> Q["Quality and SLO gate"]
    Q --> C["Conditional conclusion"]
    C --> H

smaller prefill chunks
  -> shorter mixed engine steps
  -> lower decode stalls
  -> better ITL goodput
  -> possibly lower prefill efficiency and worse TTFT

This prediction determines the measurements. If you record only total tokens per second, you cannot test it. The chain also names its own confounders: the last line predicts a cost, so the experiment must measure TTFT even though the hypothesis is about ITL — a benchmark that only instruments its hoped-for effect is an advertisement, not an experiment.

Also write a falsification condition. If smaller chunks lower ITL but reduce goodput after TTFT constraints are included, the proposed configuration did not achieve its goal. Decide the threshold now — “goodput must improve by at least 5 percent or we keep the simpler configuration” — because a threshold chosen after seeing the data will always be met by something. Pin the analysis revision with the hypothesis too: which script, over which raw events, computes the verdict. A result whose analysis can drift after collection is not yet an experiment.

Use the right benchmark level

A kernel microbenchmark isolates one operation and is ideal for comparing implementations across shapes. An engine-step benchmark includes input preparation, kernels, graphs, and collectives. An end-to-end load test includes queues, routing, preprocessing, streaming, and clients.

Each level removes noise and context. Start at the lowest level that can answer the question, then verify at the service boundary. A kernel speedup that vanishes at the engine step is still useful diagnostic evidence; it is not an end-to-end performance result.

The engine-step level, walked

The engine level earns its keep because step composition is where most scheduler claims live, and it is cheap to instrument. Using Atlas’s frozen costs: a pure-decode step at the operating batch runs 45 ms. A mixed step that admits a 256-token prefill chunk pays roughly 20 + 0.035 × 256 ≈ 29 ms of prefill work (the frozen prefill model, applied to a chunk), so the mixed step costs on the order of 45 ms of decode work plus that chunk — decode tokens in that step see their ITL stretch by however much the chunk extends the step. That arithmetic is the whole hypothesis of chunked prefill in miniature: the chunk size dial trades prefill throughput against a per-step ITL tax that the service-level benchmark will later either forgive (ITL budget 150 ms has room) or punish (tail ITL already near budget). Running the sweep at the engine level first costs minutes; discovering the tax at the service level costs a full load-test cycle.

What the engine level cannot tell you is equally important: queueing, admission, routing, and client behavior are absent by construction. An engine result is a component input to the service decision, never the decision.

Microbenchmarks and their traps

Below the engine level sit kernel microbenchmarks, and they have failure modes of their own worth naming because vLLM’s pinned tree dedicates whole directories to getting them right — benchmarks/kernels and fused_kernels, plus an overheads directory for framework-level fixed costs. Four traps account for most wrong microbenchmark numbers:

• Warm caches measure a different kernel. Repeating one shape back-to-back keeps weights and activations resident in levels of the memory hierarchy that a serving step — interleaved with other layers’ traffic — never enjoys. Rotate shapes or flush deliberately.
• Clock resolution versus kernel duration. Timing a 50-microsecond kernel with coarse host clocks, or through launch overhead comparable to the kernel itself, measures the launcher. Use device-side timing or amortize over many launches and say which you did.
• Shape cherry-picking. Reporting the three shapes where your kernel wins, from a sweep of thirty, is a marketing document. Publish the full sweep; Chapter 9’s capture-signature discipline exists partly so the served shapes are enumerable and therefore benchmarkable.
• No error accounting. A fused kernel that runs faster while changing accumulation order has changed the product. The layer’s own table demands “latency and numerical error” — treat max-abs and distributional deltas as first-class outputs, not footnotes.

Estimating capacity before measuring

Every serving team asks some version of the same question in a design review: roughly what will this configuration sustain? Benchmarking answers it eventually; arithmetic answers it now, well enough to know whether the plan is plausible. Walk the whole estimate for one Atlas TP4 island using only frozen and declared numbers.

Decode throughput. A model step reads the 35 GB weight shard plus each active sequence’s state. Against a declared memory path of ~3 TB/s, weights alone imply 35 / 3 ≈ 12 ms of streaming; adding 32 sequences at 2,000 context tokens (32 × 2000 × 80 KiB ≈ 5 GiB, about 1.7 ms more) sets a roofline floor near 14 ms. Atlas’s declared step is 45 ms — about a third of the roofline — which is what real engines cost once launch overhead, attention addressing, sampling, and collectives join the streaming. Take the 45 ms as given: throughput is 32 tokens / 0.045 s ≈ 700 output tokens per second per replica at batch 32.

Concurrency. KV capacity caps resident sequences before speed does: 35 GiB / 0.61 GiB per 8,000-token sequence ≈ 57, so batch 32 sits at 56 percent occupancy — headroom, not accident, given Chapter 6’s preemption arithmetic.

Cross-check with utilization. The same step in arithmetic terms is 2 × 70 GFLOP × 32 ≈ 4.5 TFLOP per 45 ms across four accelerators of ~1 PFLOPS-class peak: roughly two to three percent MFU, exactly the decode ceiling Chapter 4 derived from intensity. And the frozen prefill cost back-solves instructively: prefill_ms(2000) implies moving about 280 TFLOP in 90 ms on the same island — near three-quarters of peak, an aggressive large-batch figure that says Atlas’s constants describe a well-tuned system. When your measured prefill MFU lands far below that, the gap is a diagnosis queue, not a mystery.

Assemble. At ~200 output tokens per response, decode alone sustains about 700 / 200 ≈ 3.5 requests per second per replica; applying Chapter 2’s operating-utilization discipline and TTFT admission trims that to a few qualifying requests per second — within a factor of two of Chapter 25’s declared operating point, which is precisely the accuracy band a pre-benchmark estimate should claim. The estimate’s real products are the constraints: concurrency bounded by KV bytes, throughput by step time, TTFT by prefill-plus-queue, and each bound naming the knob that would move it. Benchmarks then refine numbers you can already defend; without the estimate, they refine numbers you cannot.

Reproduce the workload

A benchmark card should record arrival process, input and output length distributions, prefix reuse, modality, priority, sampling parameters, concurrency, and cache state. Preserve important correlations — real traffic couples input length to output length (long documents get long answers) and couples arrival bursts to working hours. A benchmark that samples each distribution independently measures a workload that exists nowhere.

Use an open-loop generator for externally driven traffic and overload studies. Use a closed-loop generator when modeling a fixed population of clients that wait before issuing more work. Label the choice.

Warm and cold tests answer different questions. A cold test includes model loading, compilation, graph capture, and empty caches. A steady-state test should define its warm-up and confirm that compilation or allocation is no longer changing the system — Chapter 9’s capture-once discipline means a well-behaved engine converges, but a benchmark that starts measuring before the last graph is captured is measuring the compiler, not the service.

Chapter 2 defines open- and closed-loop traffic and derives why they expose different overload behavior. A benchmark uses that distinction rather than re-deriving it: choose open loop when arrivals are externally driven or overload is under test, and closed loop only when a fixed client population is itself part of the workload.

What a trace replay must preserve

Trace replay — SGLang’s use_trace_timestamps mode, scaling recorded arrival times by a slowdown_factor — is the most faithful generator and has its own validity conditions. The replayer must actually keep up: if issuing a request takes longer than the next trace timestamp allows, arrivals silently compress and the offered load drifts above the value on the card. Assert on issued-versus- scheduled timestamps rather than trusting the sleep loop. Prefix-cache state carries across requests within a run, so replay order is part of the workload — shuffling a trace changes reuse even though its length distribution is untouched. Multi-turn conversations are another preserved correlation: a follow-up turn’s prompt includes the prior answer, so breaking conversations into independent requests changes both prefix structure and input-length distribution at once. vLLM’s benchmark package keeps a dedicated multi_turn directory for exactly this reason. Finally, record whether the harness pinned sampling (temperature: 0.0 in vLLM’s default payload) or replayed production sampling parameters — the first isolates system performance, the second is more faithful to output-length variance, and the card should say which.

Measure latency-bounded throughput

Increase offered load until one or more SLOs fail. Plot goodput rather than publishing one throughput point. The curve reveals saturation and collapse.

Offered load must be swept through the service’s operating regimes.

flowchart LR
    L["Low load"] --> S["Saturation approach"]
    S --> O["Overload"]
    L --> M1["Latency floor"]
    S --> M2["Goodput knee"]
    O --> M3["Queue growth and rejection"]

Learn to read the curve’s three regions. At low load, goodput tracks offered load one-to-one and latencies sit near their floor — the system is transporting, not queueing. Near the knee, goodput growth flattens as queue delay consumes SLO budgets; this is the operating region, and production should sit below it with headroom sized for arrival bursts. Past the knee, goodput falls as offered load rises — queues lengthen, TTFT breaches spread, and every additional request makes the others worse. The single most useful number from a sweep is not peak goodput but the distance between the chosen operating point and the knee, because that distance is the service’s tolerance to a traffic surprise.

Report latency distributions, not only averages. Keep errors, cancellations, and timeouts in the accounting. A request that disappears from the sample when it times out makes the service look better as it fails — the timeout filter is a machine for converting overload into flattering numbers.

Benchmark standards such as MLPerf Inference demonstrate the value of defined scenarios, quality targets, and run rules for comparable results. Your production benchmark will use different models and traffic, but it should be equally explicit about the contract.

Inside an open-loop harness

The pinned serving benchmarks show how much contract hides inside a load-generator script. vLLM’s vllm/benchmarks/serve.py at the pinned SHA generates arrivals in get_request: with burstiness = 1.0 intervals are exponential — “it follows exponential distribution” — and the general case samples from a gamma distribution whose shape is the burstiness parameter, so one dial moves arrivals between bursty and uniform. Two details reward attention. First, the generator precomputes the whole arrival schedule, then rescales it: the comment notes that summed gamma draws “would have 1-2% gap from target_total_delay_s,” and normalization “close[s] the gap for stabilizing the throughput data from different random seeds” — even the arrival process is calibrated, because an unnormalized generator would report throughput variance that belongs to the harness, not the system. Second, self_timed mode abandons synthetic arrivals entirely and replays recorded trace timestamps, scaled by a slowdown factor — the workload is the arrival process, so the most faithful generator is the one that stops generating.

SGLang’s sglang/benchmark/serving.py adds a measurement-validity correction this book’s Chapter 11 makes predictable: with speculative decoding, one streamed chunk can carry several accepted tokens, so raw per-chunk inter-token latencies overstate per-token latency. Its use_retokenized_itl path divides each chunk’s ITL by the retokenized token count of that chunk’s text (adjusted_itl = itl / num_tokens) and expands the series accordingly. A harness that ignored the bundling would “measure” speculative decoding as an ITL regression.

Both harnesses agree on what counts as a request. In vLLM’s backend_request_func.py, a stream that returns HTTP 200 but never delivers a token-bearing chunk is recorded as failed — “Never received a valid chunk to calculate TTFT. This response will be marked as failed!” — and the payload pins temperature: 0.0, because sampling variance is not what is under test. Goodput is a per-request conjunction: is_good_req = all(...) over every configured SLO, so a request that meets TTFT but misses TPOT counts as failed for goodput purposes. Chapter 22’s rule — “both returned HTTP 200 is not conformance” — is the same discipline pointed at correctness; here it is pointed at speed.

Protect semantic equivalence

Two systems are comparable only if they perform equivalent work. Check model weights, precision, tokenizer, template, context limit, sampling, stop rules, structured output, and output quality.

Quantization and speculative decoding require quality or distribution checks. Prefix caching requires output equivalence. Different truncation policies can make one engine appear faster by processing less input. Record accepted and rejected speculative tokens separately from user-visible output — the retokenized-ITL correction above is exactly why: the acceptance statistics are the mechanism’s evidence, while the retokenized series is the user’s experience, and collapsing one into the other loses both.

When strict equality is not expected, define the evaluation and acceptable change before seeing the result. “Within 1 point on the quality task” chosen after the run is not a gate; it is a rationalization with a number attached.

Design the gate to match the mechanism’s risk. Greedy decoding under an algebraically identical kernel should match near-exactly, with divergence only from floating-point non-associativity — so the gate is a tolerance on token divergence position, not a task score. Quantization changes the product by design, so the gate is a task-metric regression bound agreed before the run, plus a distributional check on the outputs Chapter 10 recommends. Speculative decoding should be output-preserving under its verification contract — any accepted-token distribution shift is a bug in the drafter or verifier, not a quality trade-off to be weighed. Structured-output changes gate on Chapter 21’s conformance suite rather than on task metrics, because the property under test is syntactic and behavioral, not statistical. One gate per risk; a generic “quality looks fine” gate catches none of them.

Control the environment

Record engine and model commits, container digest, compiler, drivers, firmware, device model, power settings, CPU and NUMA placement, interconnect topology, and relevant environment variables. The test of adequacy is rebuildability: a reader with the card should be able to construct a byte-identical system. A version string does not satisfy this — the same release with different launch flags is a different system, and flag drift between “identical” runs is a classic source of irreproducible comparisons. Note other workloads on shared hardware — a neighbor’s training job can move your p99 by more than the optimization under test.

Run enough repetitions to characterize variance. Randomize experiment order when temperature or shared infrastructure can drift: thermal state is a slow variable, and a fixed A/B-A/B order lets it masquerade as a treatment effect. Report confidence intervals or the raw distribution instead of excessive decimal precision.

Do not tune one system extensively while leaving another at defaults. Either compare documented defaults for a stated purpose or give each system a fair tuning budget and publish the configurations.

How many repetitions, and of what

Repetition budget follows from the statistic being claimed, and the demands are not symmetric. A mean stabilizes quickly; a tail does not. Estimating p99 latency within a useful tolerance requires observing many samples beyond it — with 100 requests, the p99 estimate is effectively “the worst request,” which is one sample wearing a percentile costume. As a declared working heuristic: tail percentiles need thousands of requests per cell to move less than the effect you are testing for, so either collect that volume or claim a lower percentile honestly. This is also why per-request raw events (the card’s outputs line) matter more than the summary: pooled across runs they support bootstrap intervals, while pre-aggregated summaries cannot be re-analyzed.

Structure the repetitions as blocks. Run baseline-and-candidate back-to-back within each block, then repeat blocks in randomized order — blocking absorbs the slow drift (thermals, background load) into comparisons within a block, where both systems see the same environment. Report the spread across blocks; if blocks disagree beyond their internal noise, the card’s second-day procedure has already told you what to inspect next.

Profile after locating the regime

A profiler explains a result; it does not define the workload. First identify the batch sizes and load range where behavior changes. Then capture CPU and GPU timelines, kernel counters, memory activity, collectives, and network traffic in that regime.

Look for idle gaps, synchronization, unexpected copies, graph fallbacks, padding, imbalance, and queue transitions. Connect every low-level observation back to a service metric. “GPU utilization increased” matters only if useful output or cost improved — utilization is a diagnostic, and optimizing it directly produces busy systems that serve no one.

Capture has a cost, and the cost perturbs the thing being measured: timeline tracing lengthens steps, and a capture run is therefore a different benchmark from the un-instrumented run that produced the headline number. Treat profiling runs as explanatory evidence attached to a regime, never as the source of the performance claim — Chapter 5’s step anatomy was built from exactly this kind of capture, and it explains the 45 ms step without being the place the 45 ms was certified.

Publish negative and conditional results

An optimization that loses under high concurrency or low prefix reuse is valuable information. It helps readers learn the boundary of the mechanism.

Use conditional language:

On this model and device, with the measured shape distribution, configuration A increased TTFT-qualified goodput by 18 percent. It regressed the low-load median because graph padding dominated below eight active sequences.

This result is more durable than a framework ranking. The 18 percent will not transfer to another reader’s traffic, but the shape of the boundary — where padding stops being amortized — will, and the next reader can test their own position relative to it.

Negative results also compound only if they survive publication: attach the card, raw events, and analysis revision to the report, because a negative result without its evidence cannot be re-tested when a reader’s conditions differ. An optimization retired on an unarchived benchmark will be re-proposed, re-benchmarked, and re-rejected within the year at full cost.

Worked example: make the claim falsifiable

“Configuration B is 18 percent faster” is not a benchmark claim. A useful claim states that B improves TTFT-qualified goodput for the Atlas document trace, under an open-loop arrival process, while passing the same quality gate. It pins the model, engine commit, container, hardware topology, workload hash, warm-up, cache state, error accounting, and analysis revision.

Walk the claim against the card in Appendix G. The SLO line — success, TTFT at most 600 ms, every ITL at most 150 ms, valid output — makes goodput computable: a request is good only if all four hold, so at an offered 8 requests per second, “goodput of 6.2” means 6.2 requests per second passed every gate. Attribution then comes from the raw events, not intuition: suppose (declared example) 1.1 requests per second missed TTFT, 0.4 missed an ITL sample, and 0.3 errored — the candidate’s TTFT mechanism is worth pursuing, the error path is a bug regardless of speed, and the ITL tail needs one more sweep near the knee before any conclusion. The workload line’s prefix distribution matters because Atlas’s routing hypothesis lives or dies on reuse: run the sweep at low and high reuse, not one blend, or the result cannot say whether routing helped the cached or the uncached population.

Run baseline and candidate in randomized order at several offered loads. Keep timeouts and errors in the denominator. If a second-day result moves, inspect temperature, clock policy, cache warmth, artifact hashes, and background traffic rather than averaging two regimes into one misleading number — the card’s method section exists precisely so the second-day run differs from the first in at most the ways it lists. Every field on the card earns its place by naming a way the comparison could silently break; a field you cannot connect to a failure mode is decoration, and a failure mode with no field is an unprotected flank.

Practice: complete a benchmark card

Write the full card for a candidate scheduler that claims higher goodput on the traces from Chapter 2. Include commands, configuration, system identity, quality checks, repetitions, raw event schema, analysis revision, and second- day reproduction procedure.

State the exact claim the evidence could falsify. A complete example appears in Appendix G.

24. Observability, Reliability, and Operations

At 14:07, time to first token rises while GPU utilization falls. The model workers report no errors. Is the cause a tokenizer backlog, a failed graph capture, a cache-transfer timeout, a network partition, or an empty decode pool?

Observability is the ability to answer that question from the system’s outputs. It begins with a model of the request path, not a large dashboard. Each candidate cause lives at a different boundary — ingress, compilation, transfer, membership, admission — so the system must expose a signal at every boundary it owns, or the first incident becomes an archaeology dig with users as the time pressure. “No errors reported” usually means “no error path was instrumented,” not “nothing is wrong.”

The discipline mirrors Chapter 23’s: a diagnostic is an experiment whose question is “which component broke,” and like any experiment it needs its measurements designed before the event.

Metrics show shape; traces show path

Metrics summarize behavior over time. Useful families include arrival and completion rates, queue age, TTFT and ITL histograms, scheduled tokens, active sequences, memory pressure, cache matches, transferred bytes, graph dispatch, preemption, and errors.

Operations needs signals from the request path and its resource owners.

flowchart LR
    R["Request path"] --> M["Metrics: rates and distributions"]
    R --> T["Traces: waits and boundaries"]
    R --> L["Logs: decisions and failures"]
    M --> D["Diagnosis"]
    T --> D
    L --> D
    D --> A["Safe action and rollback"]

Prefer queue age to queue depth as the headline signal — Chapter 21 made the same discovery for realtime media. Depth conflates arrival rate with service rate: forty queued requests during a healthy 45 ms-step regime is a normal instant; forty queued requests whose oldest member has waited two seconds is an incident in progress. Age is directly comparable against the TTFT budget (how much of the 600 ms has the oldest request already spent?), survives changes in batch composition, and degrades gracefully when request sizes are heterogeneous. Record both if storage allows, but alert on age.

Logs record discrete decisions and failures. They should include request or operation identity, component, state transition, version, and reason without leaking prompt content or secrets.

Distributed traces follow one request across the router, preprocessing, engine, stages, transfers, and output stream. A trace should distinguish waiting from execution. The OpenTelemetry semantic conventions provide common naming principles for traces, metrics, logs, and resources, including HTTP and RPC operations.

Use stable low-cardinality dimensions for metrics. Model, route, status, and SLO class are often useful. Request ID, prompt hash, and tenant IDs belong in traces or controlled logs; placing them in metric labels can overwhelm the monitoring system and create privacy risk.

Cardinality is a correctness constraint

Label choices have arithmetic consequences. Suppose a TTFT histogram with 40 buckets, crossed with route (5), SLO class (2), model (2), and — the tempting mistake — tenant ID (say 500 active tenants). That is 40 × 5 × 2 × 2 × 500 = 400,000 series per metric per replica; across eight replicas and six such metrics, nearly twenty million active time series for one signal family. Most tenants’ series go stale between scrapes anyway, so the store churns creating and expiring them — monitoring cost grows with traffic mix rather than traffic volume, and queries that used to scan one series per route now merge thousands. The privacy risk compounds it: tenant IDs in labels leak who your customers are to everyone with dashboard access.

The working pattern: aggregate aggressively in metrics (per route and SLO class, where capacity decisions live), and answer per-tenant questions from traces or controlled logs where each record carries identity by construction. If a per-tenant metric is genuinely required, make it an explicit allowlist of large tenants, not an unbounded label.

Trace volume needs its own policy, and it is the mirror image of metric cardinality: metrics must aggregate up front because they are unbounded over time, while traces can be sampled because each one is individually complete. Tail-based sampling keeps every trace that ended in error and every trace slower than a threshold — precisely the ones diagnosis will ask for — and samples the boring majority down to whatever storage supports. Head-based sampling is cheaper and simpler but discards slow traces because they are slow, hiding exactly the population the symptom table above routes on. Declare the choice on the card; “we have tracing” without a retention statement usually means the interesting traces were dropped first.

Alert on symptoms, not causes

Alerts should fire on user-visible symptoms — SLO burn — and leave cause hunting to humans with dashboards. Cause-based alerting (“cache hit rate below 60 percent,” “queue depth above 50”) pages on conditions that may not matter today and misses the ones that do; symptom-based alerting needs only the SLOs you already publish. The standard mechanism is multi-window burn rate: compare error-budget consumption pace over a short window against a long one, and page when both are elevated. Walked with Atlas numbers: a 99.9 percent availability target leaves 0.1 percent of 30 days ≈ 43 minutes of error budget per month. Paging when a one-hour window burns at 14 times the sustainable pace — confirmed by a slower window so blips do not fire it — means that, if nothing changed, the month’s budget would be gone in 30 / 14.4 ≈ 2 days: early enough to act, late enough that ordinary variation never wakes anyone. The same structure applies to latency SLOs: TTFT-goodput is the availability metric, its budget shrinks with every breaching request, and burn-rate paging works unchanged. Every page should link the dashboard whose queries are the first runbook branches; a page that does not start the diagnosis has wasted its most expensive resource — a human’s attention at 03:00.

Observe the scheduler and state

GPU utilization alone cannot explain an inference engine. Record the waiting and running request counts, oldest queue age, step token composition, prefill chunks, decode batch size, admission rejection, and preemption.

For memory, record free and reserved blocks, allocation failure, fragmentation or tail waste, live versus reusable state, and deferred release. For distributed caches, include lookups, matched tokens, transfer duration, cancellation, write-back, and stale location failures.

For MoE, record tokens per expert and per rank, dispatch and combine duration, stragglers, and placement generation. For disaggregation, expose every stage queue and transfer boundary. These metrics translate the architecture into operational evidence: a Chapter 18 encoder queue age answers “is vision input the bottleneck” in one glance; a Chapter 20 weight-version gauge confirms every rank serves the same policy before you blame the model.

Inside an engine’s statistics layer

vLLM’s vllm/v1/metrics/stats.py at the pinned SHA shows how much measurement philosophy fits in one dataclass file. Cache hit rate is computed by CachingMetrics as a sliding window over the most recent requests — a deque of (requests, queries, hits) trimmed to a cap, defaulting to 1,000 requests — not a lifetime average, because a lifetime average hides exactly the events operators care about: a cache flush, a traffic-mix shift, an adapter rollout all move the recent rate long before they dent the cumulative one. Its comment “DO NOT append empty stats to avoid helpful info get[ting] kicked out” records a real bug class: empty updates would dilute the window and silently drag the hit rate toward zero.

Eviction appears not as a counter but as events: KVCacheEvictionEvent carries lifetime_seconds, idle_seconds, and a tuple of reuse_gaps_seconds per block. That is eviction-policy evidence in recordable form — if blocks are evicted idle-for-minutes and then requested seconds later, the retention policy is wrong in a way no hit-rate scalar would localize. SchedulerStats separates prefix_cache_stats from connector_prefix_cache_stats, keeping local hits distinct from distributed-cache hits so a Chapter 15 connector degradation cannot masquerade as a local-cache problem.

Two more details repay study. RequestStateStats keeps timestamps in two domains on purpose: arrival_time is an “engine frontend timestamp (wall-clock),” while queued_ts, scheduled_ts, first_token_ts are “engine core timestamps (monotonic)” — the same dual-clock discipline Chapter 21 needed for media playout, applied to latency accounting, so cross-domain subtractions are explicit rather than accidental. And SchedulerIterationDetails carries an is_dummy flag alongside context-versus-generation token counts: Chapter 14’s participation steps surface as a metric field, letting a dashboard separate real work from collective-synchronizing filler — the same distinction Chapter 23 demands of any honest throughput claim.

Readiness is a sequence of states

A process can be alive before it is ready. Model download, weight load, distributed initialization, kernel compilation, graph capture, cache registration, and router membership may all need to finish before traffic is safe.

Readiness progresses through model-specific startup stages.

flowchart LR
    P["Process alive"] --> W["Weights loaded"]
    W --> G["Distributed groups ready"]
    G --> C["Kernels compiled and graphs captured"]
    C --> H["Health execution passed"]
    H --> R["Router membership ready"]

Liveness asks whether the process should be restarted. Readiness asks whether it should receive new work. A worker draining old requests is live and not ready for new ones. A worker blocked in a failed collective may have a running process and be unable to make progress — Chapter 14’s participation requirement means a hung collective looks like a paused engine, not a crashed one, so neither probe type catches it alone.

Health checks should test the dependency appropriate to their purpose. An HTTP ping to the frontend does not prove the model group can execute. A full model request can be too expensive for a frequent liveness probe.

Make the startup stages themselves observable: publish each transition in the readiness diagram — weights loaded, groups joined, graphs captured, health execution passed — as a timestamped event or gauge, so “the replica is stuck” becomes “stuck at graph capture for six minutes,” which names the component and often the fix before anyone logs in. Chapter 9’s explicit capture signatures make this natural: the set of captured signatures is readiness state, and reporting it costs one gauge.

Choosing probes, and pricing their lies

Read the rightmost columns as the probe’s blind spot. The dangerous failure is not choosing the weak probe — it is asking a weak probe a strong question: port checks answering “can this replica serve,” or a single-shape health execution answering “all captured graphs work.” Chapter 9’s explicit capture signatures give the staged gate something concrete to report; a health execution exercises one signature, so readiness should require the set, not sample it. Price matters too: a full forward pass every five seconds steals a batch slot from paying traffic on every replica — at Atlas’s 45 ms step, a health request landing each interval is a permanent tax of roughly one step in a hundred at modest load. Frequent cheap probes plus infrequent expensive ones beats one probe asked to do both.

Treat the serving image as a measured artifact

“Same model” does not mean same service. A deployment is the combination of weights, tokenizer, model code, engine revision, kernel libraries, accelerator runtime, driver, configuration, and compiled artifacts. Pin and record that combination as one release identity — Chapter 22 made template and parser revisions part of served behavior, and operations extends the same identity to everything below them.

Build containers from reproducible inputs and keep model artifacts outside the mutable container layer when their size or access policy demands it. Verify checksums before a worker becomes ready. Do not download unpinned executable model code during startup. Produce a software bill of materials and scan both the base image and Python or native dependencies, while recognizing that a clean vulnerability scan does not prove model safety.

Startup time is operational capacity. Measure image pull, model fetch, weight load, distributed initialization, compilation, graph capture, and warm-up individually. If a worker takes twelve minutes to become ready, an autoscaler cannot rescue a two-minute traffic spike. Warm pools, local artifact caches, or forecast scaling may be required — and the warm-pool size is arithmetic, not vibes: if demand can double within two minutes and a replacement worker needs twelve, the pool must already hold enough ready workers to absorb the entire spike, because none of the reactive capacity arrives in time. Twelve-minute startup converts elasticity from a control loop into a procurement decision.

Scale to zero, honestly priced

The logical extreme of elasticity is serving nothing when traffic is absent and paying only for what runs — attractive for bursty internal tools and multi-tenant platforms, and it lives or dies on the readiness sequence above. What can actually be made cheap? Weights are the bulk: pre-staged on local disk and warmed into the page cache, they load in seconds rather than minutes — Chapter 20’s sleep levels already priced the extreme at a ~3 s host snapshot against a multi-minute cold envelope. Compilation is next: captured graphs and tuned kernels serialize as artifacts (Chapter 9) if their capture was deterministic in shape set and environment, restoring in seconds; recapture costs minutes. Distributed setup and health execution take seconds more. The floor for a large model is therefore tens of seconds to low minutes even with everything staged — dominated by weight streaming and collective bring-up — and nothing about the KV cache survives, because session state cannot be snapshotted into an artifact.

That last clause sets the product contract. Scale-to-zero serves cold-start tolerant traffic: batch jobs, scheduled workloads, tenants whose first request may be slow but whose tenth is warm. Interactive traffic needs the warm-pool arithmetic above instead, and the honest comparison is per tier: pool cost per hour versus lost-or-delayed requests during ramp. An interviewer probing “why doesn’t everyone scale to zero?” wants exactly this split — what stages compress to seconds, which one dominates what remains, and why the state that makes inference good is precisely the state that cannot ride in the artifact.

Promote the same immutable artifact through staging and production. Environment configuration may change endpoints and capacity, but rebuilding between stages removes much of the evidence gathered by the canary. Store the release identity on every trace so an output or latency regression can be tied back to the exact execution environment.

Overload should fail deliberately

When queues exceed the service’s ability to recover within the SLO, reject or shed work before the deployment collapses. Preserve capacity for health, cancellation, and high-priority traffic. The rejection itself must be priced like Chapter 17’s admission veto: early and loud beats late and silent, and a shed request returns an actionable class (limit, overload, deadline) so callers can respond intelligently instead of retrying into the collapse.

Graceful modes may reduce maximum output, disable expensive optional features, route to a smaller model, lower media quality, or pause background work. Each mode needs a product and correctness contract — Chapter 21’s degradation ladder is the realtime instance, but batch endpoints deserve the same pre-agreed answers to “what may we stop doing.”

An error budget connects reliability targets to change velocity. Track failures caused by overload separately from model validation, dependency failure, and internal bugs. They need different remedies: overload wants admission and capacity work, validation wants gates in the release pipeline, dependency failure wants isolation and fallbacks — and spending the budget on the wrong category buys nothing.

Test failures on purpose

Kill one worker in a tensor-parallel group. Partition a cache from its metadata service. Delay a KV transfer. Exhaust host memory. Return a late completion after cancellation. Corrupt a downloaded model artifact in a staging environment.

For every test, observe detection time, user impact, cleanup, retry behavior, and recovery. A failover that restores traffic while leaking blocks will cause a second incident later — the pass criterion includes the Chapter 22 assertion that KV references return to baseline, not merely that errors stopped.

Detection time has an architecture-implied bound worth checking in the drill. If router membership uses a lease with a 5-second renewal period, a dead tensor-parallel rank should leave routing within roughly one lease period after its group stalls — Chapter 17’s membership epochs are the fence. User impact should then be bounded by drain behavior: reusing Chapter 20’s handoff arithmetic, draining 400 running requests at one 45 ms engine step each takes about 18 seconds, so a clean failover of a loaded replica costs roughly that long of elevated latency on the affected route. If the drill shows 4 minutes of errors instead of 5 seconds of detection plus 18 seconds of drain, the gap is the finding — something between the lease and the router is not propagating membership, and no amount of capacity would have fixed it.

Disaggregated systems deserve coupled tests. If the decode pool fails while prefill remains healthy, admission should stop before completed KV state piles up. If the remote cache fails, the service may degrade to recomputation rather than becoming unavailable — at Chapter 17’s prices, recomputing a cached 4,000- token prefix costs about 4,000 × 0.06 = 240 ms of rework, which is the number that decides whether cache-loss degradation meets the TTFT budget or should shed load instead.

Deploy without mixing incompatible state

A rolling deployment needs a model and engine compatibility boundary. Drain requests before replacing workers that own nonmigratable state. Keep cache and artifact namespaces separate across versions. Do not send a live session to a new tokenizer or weight version without an explicit migration — Chapter 20’s weight transactions and Chapter 17’s cache-version bumping are the mechanisms; deployment orchestration is what must refuse to bypass them.

Canary traffic should represent the shapes and features most likely to expose problems: long context, structured output, multimodal input, adapters, and distributed modes. Compare output correctness and goodput, not only error rate. Because every trace carries the release identity (the artifact section’s requirement), canary evaluation becomes a join rather than an inference: group traces by release, compare TTFT-goodput and conformance results per group, and a regression names its version instead of a time window that two changes share.

Rollback must remain possible after caches, schemas, or control-plane metadata change. Test it before the incident.

Write runbooks around hypotheses

A useful runbook starts from a symptom and branches on evidence.

For high TTFT with low GPU utilization, check ingress and tokenizer queues, prefill admission, cache-transfer waits, graph warm-up, and worker readiness. For high ITL with normal TTFT, inspect mixed prefill chunks, decode batch size, collective stragglers, output processing, and session transport.

Each step should name the metric or trace, expected range, safe action, and rollback. Avoid instructions that say “restart the service” without identifying which state will be lost. A branch whose action destroys the evidence that selected it — restarting workers before recording queue ages and admission reasons — converts an explainable incident into folklore.

Reversibility is the quiet requirement behind every safe action: each runbook step should state not only its rollback but when to exercise it (“remove the temporary capacity after the admission queue drains,” not “eventually”). An irreversible action can still be correct — failing over a dead replica destroys its local cache state by design — but then the runbook owes the reader an explicit list of what was destroyed and what must re-warm, which is exactly what Appendix G’s drill dashboard measures. Post-incident review closes the loop: every real incident should end by adding or correcting one branch, one expected range, or one missing metric, so the runbook converges on the system’s actual failure modes rather than its imagined ones.

Worked example: high TTFT, low GPU use

p95 TTFT rises from 480 ms to 1.4 seconds while GPU utilization falls from 72 to 38 percent. Take the two numbers together before acting: falling utilization means accelerators are starving, not saturating, so adding capacity treats a symptom and hides the cause. The Appendix G runbook branches on evidence, each step naming its confirming signal.

First split: ingress and tokenizer queue age. If those queues are deep, the engine is innocent — route around or scale that tier, and rollback is simply removing the temporary capacity once the backlog drains. If they are normal, inspect engine admission age and reasons: a surge in remote-cache waits points at a Chapter 15 dependency, not insufficient GPUs. Third, compare scheduled prefill tokens against graph-fallback and compilation events — a new shape compiling mid-incident argues for stopping canary traffic on that route, retaining the old artifact, rather than touching workers at all. Only then do readiness and collective health enter, and a failed group leaves routing before any restart, preserving live state where possible.

Suppose the cause is the failure drill’s 500 ms KV-transfer delay. Correct behavior is bounded transfer waiting, then conditional recomputation — about 240 ms per affected 4,000-token prefix, per the arithmetic above — or early rejection when even that breaches the remaining TTFT budget. Restarting workers first destroys the cached state and the evidence while leaving the dependency failure untouched. The dashboard that makes this diagnosable shows stage queue ages, transfer counts and bytes, timeout reasons, recomputation counts, and end-to-end goodput — every field mapping to one branch above.

Practice: write and test the runbook

Build a dashboard for the Chapter 15 pipeline and inject 500 ms into KV transfers. Write the high-TTFT/low-utilization runbook with expected metric ranges, safe actions, and rollback at every branch.

Measure detection, user impact, cancellation cleanup, recomputation, leaked blocks, and recovery. Give the runbook to an engineer who did not build the system. The worked branch structure is in Appendix G.

The final chapter brings the technical choices together with cost, security, and organizational ownership.

25. Economics and Architecture Decisions

The fastest configuration is not always the one a team should deploy. It may require scarce hardware, duplicate too many weights, expose administrative interfaces, or cost more per useful answer. Architecture is the process of making those constraints explicit.

This chapter joins the technical threads into economic and architectural decisions. Chapters 5 through 22 built mechanisms and contracts; Chapter 23 made their claims testable, and Chapter 24 made their failures diagnosable. Here the limiting quantity is qualifying work per dollar rather than tokens per second. Chapter 26 then applies the same explicit-boundary discipline to trust, isolation, and retained data.

Choose an economic unit that reflects value

Cost per GPU-hour is an input, not an outcome. Cost per request ignores sequence length. Cost per output token ignores quality, retries, and latency failures.

Technical efficiency becomes product economics only after qualification.

flowchart LR
    H["Hardware and service cost"] --> C["Available capacity"]
    C --> T["Completed tokens or media"]
    T --> G["SLO and quality-qualified output"]
    G --> V["Product value"]
    O["Engineering and on-call cost"] --> H
    F["Failure and idle capacity"] --> H

A stronger unit is cost per qualifying request or cost per good output token. It includes only work that meets the quality and service contract introduced in Chapter 2.

unit cost = total serving cost / qualifying work

Total cost includes accelerators, CPUs, host memory, storage, network transfer, reserved but idle capacity, software and operations, and failed or repeated work. For owned hardware, include depreciation, power, cooling, support, and the cost of capacity that cannot be reassigned.

A denominator this powerful invites gaming, so pin what counts as qualifying. If the quality gate is loosened, latency failures vanish and unit cost “improves” with no engineering at all; Chapter 23’s rule — define the evaluation before seeing the result — applies at the finance boundary too. The qualifying-work definition belongs in the ADR next to the SLOs it derives from, changed by the same review process, so that a cost improvement claim is always a claim at a fixed contract.

Compare steady and bursty workloads. A design with excellent saturated efficiency may be expensive at the product’s normal utilization.

Pricing Atlas per qualifying request

Walk the formula with declared numbers in Atlas’s own units. Assume an eight-accelerator node costs $30 per hour all-in — depreciation, power, cooling, network, and the fraction of host and control-plane cost attributed to it. At its operating point below the goodput knee, suppose the node sustains 5 qualifying requests per second under Atlas’s TTFT and ITL gates. Unit cost is 30 / (5 × 3600) ≈ $0.0017 per qualifying request — about a sixth of a cent. Now the product runs at 40 percent utilization, as consumer traffic does: the node still costs $30, but delivers 5 × 0.4 = 2 qualifying requests per second on average, so realized unit cost rises to about $0.0042 — two and a half times the saturated figure without any engineering change. This is the gap the denominator hides: procurement decks quote saturated efficiency, finance pays utilization-weighted cost, and the difference is why burst handling (Chapter 17’s admission, Chapter 24’s warm pools) is an economic mechanism, not an operational nicety.

The same walk exposes what optimization is worth. A scheduling change that lifts qualifying throughput 18 percent cuts saturated unit cost by roughly 15 percent (1/1.18) — but if it also adds a replication tier whose cost is 10 percent of the node budget, the net is near zero. Every performance claim from Chapter 23 converts to this currency before it competes for engineering time; that conversion, not the benchmark, is what a roadmap meeting needs.

TCO worked example: self-hosted versus managed API

Walk the comparison with concrete numbers to see where the breakeven lives. These are illustrative; substitute your actual costs.

Self-hosted (8× H100 node, reserved instance):
  Hardware:          $30/hr ($21,600/month)
  Engineering:       ~$5,000/month (fractional SRE, on-call)
  Networking/misc:   ~$1,500/month
  Total:             ~$28,100/month

  Qualifying throughput at 60% utilization: 3 req/s average
  Monthly qualifying requests: 3 × 3600 × 24 × 30 ≈ 7.78M
  Cost per qualifying request: $28,100 / 7.78M ≈ $0.0036

Managed API (priced per million tokens):
  Assume $3 per million input tokens, $15 per million output tokens
  Average request: 1,000 input + 200 output tokens
  Per-request cost: (1000 × $3 + 200 × $15) / 1M = $0.006

  Monthly cost at 7.78M requests: $46,680

At this volume and utilization, self-hosted costs roughly 60% of the managed API. But the picture reverses at low utilization:

Self-hosted at 15% utilization (nights, weekends):
  Same $28,100/month, 1.94M qualifying requests
  Cost per qualifying request: $28,100 / 1.94M ≈ $0.0145

Managed API at same volume:
  1.94M × $0.006 = $11,640/month

The crossover depends on sustained utilization, engineering cost, and burst headroom. Most teams start managed, switch when utilization consistently exceeds 40–50%, and keep a managed overflow route for bursts that exceed self-hosted capacity. Chapter 17’s admission control makes the routing decision explicit rather than implicit.

Utilization can hide stranded resources

An MoE deployment may show high network use and low expert compute. A disaggregated service may have a full decode pool and idle prefill GPUs. A model can consume nearly all HBM while leaving arithmetic units underused.

Report utilization by resource and stage. The limiting resource determines capacity; the others may be stranded. Independent scaling helps only when pool sizes can track the workload without adding excessive transfer or warm-up cost.

Disaggregation makes stranding concrete because Chapter 18’s stage table prices each pool separately: if prefill completes in a burst around each arrival wave while decode drains steadily, the prefill pool’s honest utilization is its busy fraction, not its peak — sizing it for the peak buys idle accelerators most of the day, and sizing it for the average converts the difference into queue age at arrivals. The same logic that set Chapter 21’s playback buffers applies to pool sizing: buffers absorb variance you predicted, not variance you didn’t.

Power limits can change kernel clocks and throughput. Energy per useful output captures a dimension that device-hour pricing may hide. If carbon-aware scheduling is a requirement, deadlines and data locality constrain when and where offline work can move.

Energy per qualifying request

Energy is unit-cost arithmetic with power substituted for rent. Assume the Atlas node draws 6 kW under serving load. At 5 qualifying requests per second, each qualifying request is responsible for 6 kW / 5 = 1.2 kJ ≈ 0.33 Wh; at 40 percent utilization the same node spreads over 2 qualifying requests per second, so energy per request rises to about 0.83 Wh — the utilization tax from the cost walk again, now in thermodynamics. Two consequences follow. First, energy per useful output is the honest carbon metric: a configuration that finishes requests faster but qualifies fewer of them can increase energy per useful output while decreasing energy per token. Second, power interacts with benchmarks — sustained load raises temperature, clocks throttle, and the tenth repetition runs slower than the first, which is why Chapter 23 randomizes experiment order and treats thermal state as a slow variable. A benchmark run that ignores its own power curve produces a number for hardware that stops existing after ninety seconds.

Managed service, self-hosted, or hybrid

A managed API transfers responsibility for engine operation and capacity while limiting control over weights, placement, and low-level optimization. Self-hosting provides control and creates responsibility for security, reliability, upgrades, and hardware supply — including everything Chapter 24 demanded: staged readiness, drill-tested failure modes, and release discipline.

Compare options using the same service contract. Include engineering and on-call cost, time to support new models, compliance, portability, failure independence, and exit cost. A lower accelerator rate can be more expensive if the team cannot keep the deployment reliable. Exit cost deserves explicit pricing because it is paid exactly when leverage is worst: migrating off a provider under deadline pressure means re-validating templates, parsers, and conformance suites (Chapter 22’s) against a new engine while production runs. Teams that priced exit discovered the conformance suite is the exit plan — portable tests convert migration from a rewrite into a rerun.

Hybrid designs may use managed capacity for bursts or selected models. The subtle requirement is semantic: an overflow route that silently changes tokenizer, template, or sampling behavior produces different answers for the same request depending on load — Chapter 23 would call that a confounded experiment running in production. Route by request class with a pinned contract per class, and measure both routes’ quality gates separately.

The sourcing comparison compresses into one table once the hidden costs are named:

Read the marginal-cost row against the Atlas walk: the managed API’s elastic pricing is exactly the utilization tax removed — you pay per qualifying work rather than per idle hour — which is why it fits bursts, and why routing baseline traffic there can cost multiples of a well-utilized self-hosted node even at similar list rates.

Write the architecture decision

An architecture decision record for an inference service should contain:

The architecture decision joins workload, placement, and trust boundaries.

flowchart TB
    W["Workload and SLO"] --> A["Architecture decision"]
    M["Model and state topology"] --> A
    H["Hardware and network"] --> A
    S["Security and data policy"] --> A
    A --> E["Benchmark and failure evidence"]
    E --> R{"Review trigger crossed?"}
    R -->|Yes| A
    R -->|No| D["Continue deployment"]

• workload distributions and growth assumptions;
• quality, latency, availability, and cost targets;
• model stages and persistent state;
• hardware and network topology;
• parallel, scheduling, cache, and routing plans;
• overload, failure, deployment, and rollback behavior;
• security boundaries and data retention;
• benchmark evidence and rejected alternatives;
• assumptions that trigger a future review.

The rejected alternatives matter. They show which constraints led to the decision and prevent a future team from repeating the same investigation without new evidence. Write them with their conditions, not just their conclusions: “TP8 rejected because wider collectives hurt interactive decode at our batch sizes” tells a future reader the rejection binds below some batch size — new evidence about batch mix reopens it, which is exactly what the review triggers formalize.

Someone must own the triggers. A review trigger that no dashboard watches and no calendar checks is prose, not governance — so the ADR names, for each trigger, the metric that measures it (already emitted per Chapter 24), the threshold, and the review cadence or event that forces a look. Quarterly review plus event-driven review on trigger crossing is a common pairing; the specific choice matters less than the property that reopening the decision never depends on someone remembering the document exists.

Worked example: a decision with triggers

Atlas begins with self-managed four-way tensor-parallel replicas, continuous batching, local prefix caching, and hybrid queue-plus-locality routing. Prefill and decode remain colocated until measured long-prompt interference repays the KV transfer boundary — Chapter 18 priced the swap: 35 ms of transfer against roughly 80 ms of recovered queue time for matched long prompts, so the trigger is a measured interference population, not a fashion. A managed API is an explicit overflow route, not an invisible retry.

TP8 is rejected because wider layer-frequency collectives hurt the interactive regime. Unconditional disaggregation is rejected because short prompts do not repay transfer. Tenant caches default to isolated, and model artifacts, administrative controls, and public generation use separate security boundaries.

The review triggers make the decision falsifiable, each naming its threshold: if context length doubles from 8,000 tokens, the Chapter 22 peak-sequence arithmetic doubles too — roughly 7.4 GiB of KV per request — and KV capacity or lower-precision KV becomes the binding constraint, reopening parallelism and cache-precision choices. If prefix reuse falls below the level where routing’s locality benefit beats its queueing cost, hybrid routing reverts to plain admission. If bursts grow shorter than worker startup — Chapter 24’s twelve-minute readiness against sub-minute spikes — warm pools stop being an optimization and become the design. And if the TTFT objective tightens enough that colocated interference breaches it at any acceptable density, disaggregation stops being conditional.

Two closing properties make this record operational rather than decorative. Its benchmark evidence is a Chapter 23 card — claim, workload hash, error accounting — so the architecture’s justification can be re-tested when conditions move instead of argued from memory. And its managed-API overflow route carries a pinned contract per request class with both routes’ quality gates measured separately, so the hybrid’s economics stay honest: overflow that quietly degrades output would otherwise book savings in the cost ledger while spending quality nobody was counting. Each trigger names the metric that fires it, so the architecture reviews itself from dashboards that already exist.

Practice: write the capstone ADR

Produce the Atlas architecture record using the workload and dense model from Chapters 2–4. Include topology, scheduling, caching, routing, overload, deployment, rollback, data retention, benchmark evidence, and rejected plans.

Change traffic, context, prefix reuse, hardware price, and SLO one at a time. For each, name the threshold that triggers review rather than merely stating that cost changes. The worked ADR is in Appendix G.

A sound architecture is not the answer to one benchmark. It is a decision whose assumptions and failure modes are visible. The last architectural question is who is allowed to cross each boundary and what data survives the request; Chapter 26 takes up that question directly.

26. Security, Isolation, and Governance

An inference service is a high-throughput interpreter for untrusted prompts, media, schemas, model artifacts, and tool results. Its performance mechanisms—shared caches, reusable adapters, remote connectors, compiled graphs, and administrative APIs—also create trust boundaries. Security therefore belongs in the architecture of the request path, not in a deployment checklist added afterward.

This chapter follows data and authority through that path. It asks what one tenant can infer about another, which inputs execute code or allocate scarce resources, which artifacts join the trusted computing base, and how deletion reaches derived state that has spread across a fleet.

Multi-tenancy needs isolation at every layer

Tenants share queues, model weights, memory allocators, caches, network links, and sometimes adapters. Quotas should cover concurrent requests, token work, media processing, cache occupancy, and expensive features—not only request rate. The Chapter 22 limit-pricing arithmetic is the template: a quota that counts requests treats a 500-token summarization and a 12,000-token analysis with media as equals, which they are not — token-work quotas approximate the real scarce quantities, KV blocks and engine-step time.

Shared compute does not imply shared identity or cache state.

flowchart LR
    A["Tenant A"] --> QA["Identity and quota"]
    B["Tenant B"] --> QB["Identity and quota"]
    QA --> S["Shared scheduler and model"]
    QB --> S
    S --> CA["Namespace A cache"]
    S --> CB["Namespace B cache"]
    S --> P["Explicit public namespace"]

Memory must be cleared or safely overwritten before reuse across trust boundaries. Cache lookup needs namespaces and authorization. Metrics and logs must not expose prompts, token IDs, or identifying hashes. Timing can also leak whether another tenant has warmed a prefix.

Noisy-neighbor controls need a defined unit. Equal requests are not fair when one tenant submits 100-token prompts and another submits 100,000-token prompts: under request-count quotas, ten equal requests let the second tenant consume roughly a thousand times the KV blocks (100,000 × 320 KiB against 100 × 320 KiB per request) and orders of magnitude more engine-step time. Token-work quotas collapse that asymmetry by construction, and estimated-time quotas — predicted cost from Chapter 17’s scoring, charged before admission and reconciled after — handle modalities where token counts understate real work, like decoded media or expensive sampling modes. No unit is final; the requirement is that the unit track something the shared infrastructure actually competes for.

What one tenant’s warm prefix tells another

Cross-tenant cache hits are a correctness question disguised as an optimization. Suppose tenants A and B both submit prompts sharing a 4,000-token document prefix. Without namespaces, B’s request hits A’s warmed blocks and skips roughly 68 ms of fetch plus the matching prefill share — and the timing delta itself discloses information: B can probe whether some document has been served recently by measuring whether its TTFT drops. That is a real channel even when no bytes ever cross tenants, and it is why Atlas defaults tenant cache namespaces to isolated: the shared-document case must opt in through an explicit sharing policy, carrying authorization with it.

Isolation has a price, and quoting it keeps the default honest. With namespaces, B recomputes the prefix — at Chapter 17’s recompute price about 240 ms of engine work for 4,000 tokens — so aggregate compute rises in proportion to how much cross-tenant overlap existed. Measure that overlap before assuming it matters; teams that enable sharing “for efficiency” without measuring routinely discover the overlap was a handful of system prompts, which can be handled with a public, non-secret shared namespace instead of weakening tenant boundaries everywhere.

The model server executes untrusted inputs

Prompts can attempt to manipulate application behavior. Media can exploit parsers. Structured schemas can consume compiler resources. Tool output can contain instructions aimed at the model. Generated code or tool calls can affect external systems if the application grants authority.

Each row deserves its own containment, because they fail differently. Media parsers run native code on attacker-controlled bytes — Chapter 18’s decode- before-admit pipeline means hostile images execute inside your service boundary, so parser choice, sandboxing, and decoded-media limits (Chapter 21’s) carry security weight beyond latency. Grammar complexity is the schema analogue of a decompression bomb: Chapter 22 priced compilation off the hot path precisely so a pathological schema costs a Future, not an engine step — but unbounded grammar size still consumes memory and compile threads, so complexity limits belong in the quota list. Tool output round-trips through the model as new input, closing a loop where an external page can instruct the model as fluently as its user can.

Treat model output as untrusted data. Validate it at the action boundary, apply least privilege, require confirmation for high-impact actions, and use idempotency for retries — Chapter 22’s separate tool-execution idempotency key is here because a replayed model response must not become a repeated wire transfer. Prompt-based defenses do not replace access control.

Availability attacks look like difficult workloads

An inference service can be denied without exploiting memory corruption. An attacker can submit inputs that are valid but expensive: maximum-length prompts, decompression-heavy media, grammars with pathological compile cost, outputs that never reach a stop condition, low-reuse adapter churn, or prefixes chosen to pollute a shared cache. These requests pass a simple schema check and consume the same scarce resources as valuable work.

Enforce the cheapest limits first. Authentication and compressed-byte limits belong before media decode; token and grammar estimates belong before engine admission; tool authority belongs at the action boundary. A limit applied only after allocation protects the response contract but not capacity. Chapter 2’s overload invariant becomes a security invariant here: once admitted work exceeds bounded service capacity, honest users experience the attack as an SLO failure.

The OWASP Top 10 for LLM Applications provides a maintained taxonomy that includes prompt injection, sensitive information disclosure, supply-chain risk, improper output handling, excessive agency, and resource consumption. The NIST Generative AI Profile places these technical risks within a broader process for governing, measuring, and managing AI risk.

Protect the inference supply chain

Model repositories can include executable custom code, serialized objects, tokenizers, templates, and native kernels. Engine plugins and JIT compilation expand the trusted computing base.

Pin model and container digests. Verify signatures or checksums. Prefer safe serialization formats. Review custom model code before enabling it. Build kernels and images in controlled environments, scan dependencies, and retain a software bill of materials. Chapter 24’s release identity is the runtime half of this: pinning is only useful if the deployment refuses unpinned inputs, which is what checksum-before-ready enforces.

The contract surfaces from Chapter 22 belong in the trusted base too. Tokenizers, chat templates, tool parsers, and grammar backends are executable artifacts that shape every response; a tampered template can redirect a conversation as effectively as tampered weights, and neither the benchmark suite nor the conformance suite catches it unless both are re-run when those artifacts change — which is exactly why Chapter 18 folded processor version into execution identity and why release identity pins all of them together. Adapter artifacts get the same treatment as model code: they are weights contributed by someone, loaded into shared memory beside the base model, so they inherit the review, pinning, and scanning requirements rather than slipping in through a side door.

Separate public inference credentials from management APIs that load models, update weights, inspect memory, run profilers, or execute collective RPCs. These capabilities can alter outputs, extract information, or deny service — Chapter 19 made weight updates transactional, and Chapter 22 refused to route them through the public network; supply-chain hygiene is the same boundary viewed from the build side. The drill list from Chapter 24 belongs here as well: a staging exercise that swaps a signed artifact for an unsigned one should end in a readiness refusal, not a warning in a log nobody reads.

Retention applies to derived state

A deletion policy that removes prompts but leaves KV blocks, encoder features, logs, traces, or benchmark samples is incomplete. Derived state can preserve information about the original input.

Map every data class through the request lifecycle:

Define retention, encryption, region, access, and deletion for each tier. Ensure backups and distributed caches honor the same model. A cache’s performance value does not override a user’s deletion right or contractual boundary — which is why Chapter 17’s cache-version bumping and namespace machinery matter beyond routing: deletion across a distributed cache is an invalidation sweep with an audit trail, not an rm.

The sweep has to reach every replica the block migrated to — Chapter 15’s transfer machinery moves KV state between nodes precisely so it survives, and survival is the problem here — plus encoder-feature caches keyed by content hash, where deleting the key mapping must also age out the stored features. Backups restore old state wholesale, so their retention model bounds every tier’s effective deletion date: a cache that honors deletion instantly but sits on a filesystem backed up for ninety days has a ninety-day deletion policy whether anyone chose it or not.

Design the trust boundaries

Draw the public request path, the model and adapter supply path, and the administrative path separately. Mark where identities change, where untrusted bytes become executable work, where state crosses tenant or region boundaries, and where an action receives external authority. A boundary is incomplete until it names authentication, authorization, limits, audit evidence, and failure behavior.

Public, supply, management, and action paths cross different boundaries.

flowchart TB
    C["Public clients"] -->|Untrusted prompts, media, schemas| I["Inference boundary"]
    A["Artifact registry"] -->|Signed models and adapters| S["Supply boundary"]
    O["Operators"] -->|Privileged control| M["Management boundary"]
    I --> W["Model workers"]
    S --> W
    M --> W
    W -->|Proposed calls| X["Action boundary"]
    X --> E["External systems"]

Keep public generation credentials separate from credentials that load models, attach storage, update weights, inspect memory, invoke profilers, or execute distributed control operations. A server that exposes both paths under one authority has made every prompt-facing parser part of its management plane.

Governance turns boundaries into evidence

Governance is the machinery that keeps a reviewed boundary from drifting. Keep an inventory that joins every production route to its model, tokenizer, template, parser, adapter policy, runtime image, data regions, owner, and approval record. That is the security view of Chapter 24’s release identity. A model alias without a digest or an endpoint without an owner is an unmanaged change surface.

Review is triggered by changed authority or data flow, not by model size alone. A new tool, remote-code model, shared cache namespace, telemetry field, region, or provider can change the threat model without changing a single kernel. Conversely, a kernel-only upgrade may use the existing review when its signed artifact, conformance, isolation, and rollback evidence all remain inside the approved boundary. Record exceptions with an owner and expiry; a permanent “temporary” bypass is an undocumented architecture decision.

Security incidents need inference-specific evidence: request and tenant IDs, admission and quota decisions, artifact digests, cache namespaces and invalidation generations, administrative operations, tool proposals and confirmations, and deletion acknowledgements. Preserve that evidence without logging raw prompts by default. Drill at least the failures the design claims to contain—cross-tenant cache probes, unsigned artifact loads, runaway schemas, stale tool results, and deletion across a restored backup—then make the failed control observable in the same dashboard that operators already use.

Worked example: a cache hit becomes a side channel

Two tenants submit a shared 4,000-token document prefix. A namespace-free cache lets the second request skip prefill, creating a measurable TTFT difference even though no cache bytes are returned. The safer default isolates tenant namespaces. A deliberately public corpus may use a separate shared namespace, but its authorization and retention policy travel with the cache key.

The performance cost is recomputation; the security benefit is that latency no longer reveals another tenant’s recent work. Measure the overlap before weakening the boundary. If only a few public system prompts are shared, publish those explicitly instead of enabling cross-tenant reuse globally.

Practice: produce a threat model and deletion proof

For the Atlas deployment, enumerate trust zones for clients, routing, model workers, caches, object storage, model artifacts, adapters, tools, and administrative APIs. For each crossing, state the identity, allowed action, resource limit, retained evidence, and fail-closed behavior.

Then delete one request containing text and media. Trace every derivative—tokens, KV blocks, encoder features, logs, traces, benchmark samples, distributed-cache replicas, and backups—and provide evidence that each tier either removed the state or expired it under a declared retention bound. Compare with the worked solution in Appendix G.

A secure inference architecture does not promise that untrusted input becomes safe because a model processed it. It limits what each identity can spend, observe, retain, and cause. The appendices that follow collect the notation, hardware and portability reference, reproducibility templates, deployment and decision checklists, terminology, source provenance, worked solutions, migration guide, and debugging playbook used throughout the book.

Appendix A. Mathematical and Systems Notation

This appendix collects the quantities used throughout the book. The formulas are estimates. Replace peak specifications with measurements when making a capacity decision.

Workload symbols

For a stable system, Little’s Law relates average concurrency, arrival rate, and average time:

Q = λ · W

The relationship is useful for checking measurements. If 20 requests arrive per second and average end-to-end latency is 2 seconds, about 40 requests should be in the system on average. It does not predict tail latency or guarantee stability. Delay depends on utilization rho = lambda / capacity: the simplest queueing model with random arrivals puts average waiting time near rho / (1 - rho) service periods, so waiting stays modest until utilization approaches one and then explodes. Capacity plans that target high utilization are buying queue delay with idle capacity they no longer have.

Latency

For request r:

TTFT(r) = time(first visible output) - time(arrival)

E2E(r)  = time(final output) - time(arrival)

TPOT(r) = (E2E(r) - TTFT(r)) / (L_out(r) - 1)

TPOT is defined only when more than one output token exists. Inter-token latency is the sequence of gaps between visible output events. Report the population, window, error treatment, and percentile method with every latency distribution.

Throughput and goodput

request throughput = completed requests / duration

output throughput  = visible output tokens / duration

request goodput    = completed requests satisfying SLO / duration

Define whether cached input, padded work, rejected speculative tokens, cancellations, and retries enter any numerator.

Model memory

A simple parameter-memory estimate is:

weight bytes = parameter count * average bits per parameter / 8

Include quantization scales, zero points, padding, embeddings, and replicated parameters. A complete device budget is:

device bytes = weights + persistent request state + activations
             + graph pools + communication buffers + allocator reserve

For a conventional KV layout:

KV bytes per sequence
  = 2 * layers * tokens * KV heads * head dimension * bytes per element

The formula must be adapted for latent attention, sliding windows, recurrent state, sharing, parallel sharding, and cache quantization.

Compute and movement

Arithmetic intensity at one memory boundary is:

I = operations / bytes moved

With peak compute C and bandwidth B:

attainable operations per second <= min(C, B * I)

For a transfer of S bytes with startup latency a and sustained bandwidth b, a first approximation is:

transfer time = a + S / b

Concurrent transfers, contention, registration, serialization, and synchronization add cost.

Parallelism

Use these dimension names consistently:

Do not infer group composition from the product of sizes. Record the rank tuple and communication groups explicitly.

Cache value

Raw hit rate is:

hit rate = cache lookups with any usable match / lookups

More useful measures include:

token reuse rate = matched reusable tokens / eligible input tokens

saved compute per byte = estimated compute time avoided / bytes retained

transfer amplification = bytes moved through cache tiers / bytes consumed

Use measured prefill time where possible instead of assuming equal cost per token.

Cost

cost per qualifying request = total service cost / SLO-qualified requests

cost per good output token = total service cost / output tokens from
                             SLO- and quality-qualified requests

State the accounting window and included infrastructure, software, labor, network, storage, power, reservation, and failure costs.

Appendix B. Hardware and Portability Reference

This appendix is a checklist for investigating a deployment. It avoids product performance tables because hardware and software support change faster than the principles in the main text.

Memory tiers

Accessible memory is not necessarily local memory. Unified addressing and coherence simplify programming while physical movement still affects latency and bandwidth.

Topology discovery checklist

Record:

• device model, memory capacity, power mode, and supported numerical formats;
• peer-to-peer connectivity and link width between every device pair;
• CPU sockets, cores, NUMA nodes, and memory attachment;
• NICs, rails, link rate, RDMA support, and device affinity;
• switch and rack boundaries, oversubscription, and failure domains;
• local storage devices and paths used for models or cache;
• driver, runtime, communication-library, firmware, and kernel versions.

On NVIDIA systems, tools such as nvidia-smi topo -m and NCCL topology logs can help reveal device relationships. Use the corresponding vendor tools on other platforms. Verify with a bandwidth and latency test; a discovered link does not prove the expected path is active.

Collective operations

Message size and synchronization determine behavior. Measure small decode messages and large prefill messages separately. Aggregate bus bandwidth does not reveal a slow rank or overloaded rail.

Communication questions by parallel method

Tensor parallelism: How many collectives occur per layer? Are they inside a fast local fabric? Can communication overlap adjacent computation?

Pipeline parallelism: What activation crosses each boundary? How many microbatches are needed to fill the pipeline? Which stage is slowest?

Expert parallelism: What is the dispatch distribution by rank? Are prefill and decode using appropriate transport modes? Which expert creates a straggler?

Context parallelism: Is KV state circulated, gathered, or reduced? How does communication grow with context? How are partial softmax statistics combined?

Disaggregation: How many state bytes move per request? Can block layout or parallel-size differences require gathering and scattering? What pins the source and destination during transfer?

Memory-budget worksheet

For each rank, fill in:

Run the worksheet at the largest legal shape and during warm-up. Peak phases can occur in a different order from steady serving.

Order-of-magnitude classes

Interviews and design reviews move faster when orders of magnitude are already in your head. These are classes, not product claims — each generation moves the boundaries — but the ratios between rows are the durable part. Treat them as declared planning figures in the book’s sense.

Two habits make the table useful rather than trivia. First, keep the ratios: intra-island bandwidth is roughly a hundred times the host path, and arithmetic peak is roughly a thousand times HBM bytes per second — those two ratios explain most of this book’s architecture. Second, re-derive crossovers rather than memorizing them: peak divided by bandwidth gives the arithmetic-intensity knee (Chapter 4), and it moves every generation even when the ratio structure does not.

Common traps

• Measuring device-to-device bandwidth without the application’s concurrent compute and message sizes.
• Mapping logical ranks in a way that sends frequent collectives across nodes.
• Ignoring CPU affinity and memory placement.
• Reserving all free HBM for KV state before graph capture.
• Assuming a quantized format has a native kernel on every device.
• Treating link bandwidth as available to every pair simultaneously.
• Placing redundant replicas in the same network or power failure domain.
• Using an average transfer size that hides many small control operations.

Platform portability

This book uses NVIDIA GPUs and CUDA as its default platform because, at the time of writing, most production inference systems run on them. But the principles in the main text are not NVIDIA-specific. Many of them — scheduling algorithms, cache management policies, routing logic, API contracts — are device-agnostic by construction. Others — kernel selection, graph capture, quantization formats — carry the same structural intent to every platform but require different implementations.

This appendix maps the book’s concepts to four non-NVIDIA platforms. It is not a product comparison. It does not rank platforms or declare winners. Its purpose is narrower: if you have understood a chapter’s principle on NVIDIA hardware, this appendix tells you where that principle transfers directly and where you need to learn a platform-specific mechanism to apply it.

The platforms covered are AMD Instinct with ROCm, Google TPU with JAX/XLA, Intel Gaudi with SynapseAI, and AWS Trainium/Inferentia with NeuronSDK. Each is a first-class target in at least one major inference framework (vLLM, SGLang, or both) and each has production deployments serving real traffic.

Transfer table

The table below covers the book concepts that have platform-specific implementations. For each concept, the “What changes” column describes the structural difference — the thing you must account for when moving between platforms.

What transfers without change

The following book concepts are implemented on the host CPU, in framework-level logic, or at the API boundary. They do not touch device-specific code and transfer to any platform without modification.

Scheduling algorithms (Chapter 6). Continuous batching, preemption policies, priority queues, and admission control are CPU-side decisions. The scheduler calls the model runner with a batch descriptor; it does not know or care what device executes the batch. A scheduling algorithm written for an NVIDIA deployment works identically on AMD or TPU hardware. The only indirect effect is that device speed changes the time budget the scheduler has per step, which may shift the optimal batch size or preemption threshold — but the algorithm itself is unchanged.

KV cache block management (Chapter 7). Block tables, copy-on-write, prefix-tree indexing, eviction policies, and the paged-memory abstraction are all host-side data structures. The block manager allocates and tracks blocks; a device-specific allocator provides the underlying memory. Swapping the allocator is a clean interface change. The management logic, which is the subject of Chapter 7, does not change.

Routing and the control plane (Chapter 17). Load balancers, session-affinity routers, cache-aware routing, replica health tracking, and failover logic are network-layer components. They operate on request metadata and backend health signals, not on device APIs. A routing policy designed for an NVIDIA fleet applies without modification to a heterogeneous fleet, provided the backends expose the same health and capacity signals.

API semantics (Chapter 22). The OpenAI-compatible API contract — streaming SSE, token-level callbacks, usage accounting, tool-call formatting — is defined at the HTTP boundary. It is identical regardless of the device behind the engine. Framework implementations (vLLM, SGLang) expose the same API surface on every backend they support.

Benchmarking methodology (Chapter 23). The measurement discipline — control variables, warm-up policy, percentile reporting, load-generation method — is statistical methodology. It applies to any system that processes requests. Only the profiling tools (the row in the transfer table above) are platform-specific; the experimental design is not.

Operational practices (Chapter 24). Health checks, graceful drain, rolling deployment, canary analysis, capacity planning, and incident response are operational patterns. They depend on the control plane and monitoring infrastructure, not on the device. An operational runbook written for an NVIDIA deployment needs only tool-name substitutions (replace nvidia-smi with rocm-smi or hl-smi) to apply elsewhere.

What requires platform-specific work

These areas share the same intent across platforms but require different implementations. When porting a deployment, budget engineering time for each.

Kernel selection and fusion

Every platform has its own kernel library, and the set of available fused operations differs. A fused attention-plus-RoPE kernel that exists on CUDA may not have an equivalent on ROCm or SynapseAI. The consequence is not just a name change — the optimal operation boundaries (which operations to fuse, which to leave separate) may differ because the available fusions differ.

On NVIDIA, kernel selection is often implicit: FlashAttention, cuBLAS, and Triton kernels are selected by the framework’s backend dispatcher. On AMD, the Composable Kernel library and ROCm’s Triton fork fill the same role, but the available tile shapes and fusion patterns may differ. On TPU, custom kernels are written in Pallas (a JAX-native kernel language) and compiled through XLA; there is no equivalent of loading a precompiled CUDA binary. On Gaudi, SynapseAI provides a fixed set of fused operators, and custom kernels use Habana’s TPC programming model.

The engineering task: for each model architecture, verify that every operation in the critical path has a performant kernel on the target platform. Measure at step granularity, not kernel granularity, because a missing fusion may shift work to a neighboring kernel in a way that isolated benchmarks miss.

Graph and compilation artifacts

Chapter 9’s framework — artifacts, warm-up cost, padding overhead, bucket selection — applies everywhere, but the artifact format and lifecycle differ.

CUDA graphs and HIP graphs are structurally similar: capture a stream of operations, replay them with updated parameters. The warm-up cost is the capture time, and the padding cost is determined by the bucket strategy. HIP graphs on ROCm follow the same API pattern and the same engineering tradeoffs.

XLA compilation on TPU is a fundamentally different model. There is no eager fallback in the serving path. Every distinct input shape triggers a compilation (or retrieves a cached compilation). The warm-up cost is compilation time, which can be significant for models with many shape variants. The padding cost is determined by the shape-bucketing strategy, just as with CUDA graphs, but the consequence of a cache miss is a full recompilation rather than a fallback to eager execution. SGLang’s JAX backend manages this by maintaining a compilation cache with shape buckets tuned for inference workloads.

NeuronSDK compilation is ahead-of-time: the model is compiled to a Neuron Executable File Format (NEFF) before serving begins. Shape changes require recompilation. This is the most constrained model — the artifact is fixed at deployment time, and runtime shape flexibility depends entirely on the bucketing strategy chosen during compilation.

SynapseAI graph compilation on Gaudi falls between these extremes. It supports recipe-based graph capture with some runtime flexibility, but the compilation cost is higher than CUDA graph capture and the shape constraints are tighter.

Quantization format support

The format-selection frontier from Chapter 10 is smaller on non-NVIDIA platforms. FP8 support, which is broad on H100 and later NVIDIA hardware, is available on MI300X and Gaudi 3 but with different kernel coverage. INT4 formats (AWQ, GPTQ) have mature CUDA kernels but may lack optimized implementations on other platforms. On TPU, quantization is typically applied through JAX’s AQT (Accurate Quantized Training) library, which supports a different set of formats than the CUDA ecosystem.

The engineering task: for each target precision, verify that (a) a kernel exists, (b) it handles the model’s shapes efficiently, and (c) the quality impact matches what was measured on the reference platform. Do not assume that a format that works well on CUDA will have equivalent kernel performance on another platform.

Communication libraries

NCCL, RCCL, HCCL, and NeuronSDK collectives implement the same collective operations (all-reduce, all-gather, reduce-scatter, all-to-all) but with different performance characteristics, topology awareness, and configuration surfaces. The mapping of logical ranks to physical devices, which Chapter 13 treats as a critical performance decision, depends on the platform’s interconnect topology.

Key differences in practice:

• Intra-node bandwidth. NVLink and NVSwitch provide 900 GB/s per direction on H100 systems. AMD’s Infinity Fabric on MI300X provides comparable bandwidth through xGMI links. TPU pods use a dedicated ICI mesh. Gaudi uses an internal mesh with 600 GB/s bisection bandwidth per node. The absolute numbers shift the crossover point between tensor parallelism and pipeline parallelism.

• Inter-node transport. All platforms support RDMA-capable networks, but the integration differs. NCCL auto-detects topology and selects algorithms; RCCL does the same on AMD systems. On TPU, inter-pod communication uses DCN (data center network) with different latency characteristics than ICI. On AWS, EFA (Elastic Fabric Adapter) provides the RDMA path for both Trainium and GPU instances, but NeuronSDK collectives are optimized specifically for EFA’s topology.

• Configuration. Environment variables, topology files, and algorithm selection differ across libraries. A deployment that tunes NCCL with NCCL_ALGO, NCCL_PROTO, and topology XML needs equivalent tuning for the target platform’s library.

Memory management APIs

Chapter 7’s block-management logic is device-agnostic, but the underlying memory allocator is not. Each platform provides its own allocation, deallocation, and transfer APIs. On NVIDIA, this is cudaMalloc, cudaMemcpy, and the CUDA memory pool. On AMD, the HIP equivalents (hipMalloc, hipMemcpy) are nearly identical. On TPU, memory management is handled by the XLA runtime and is largely invisible to the application. On Gaudi, SynapseAI manages device memory through its own allocation API. On Trainium, NeuronSDK handles memory layout during compilation.

The practical consequence is that memory-pool tuning, defragmentation strategies, and the interaction between graph pools and KV cache pools (discussed in Chapters 7 and 9) require platform-specific configuration even when the management policy is identical.

Profiling toolchain

The methodology from Chapter 23 is universal, but the tools that implement it are not interchangeable. Each platform’s profiler exposes different levels of detail, different visualization formats, and different overhead characteristics.

When porting a performance investigation, map each measurement from the original profiling tool to the corresponding capability on the target platform. Not every measurement has a direct equivalent — for example, warp-level occupancy analysis (Nsight Compute) has no direct analog on TPU, where the execution model is fundamentally different.

Practical guidance

Start with what transfers. When evaluating a new platform, begin with the device-agnostic layers: scheduling, cache management, routing, and API compatibility. These are the largest fraction of the system by code volume and they work immediately. The platform-specific layers — kernels, graphs, quantization, collectives, profiling — are fewer components but require deeper investigation.

Measure, do not assume parity. ROCm reaches 90–95% of H100 throughput for standard inference workloads in mature frameworks, but this is an aggregate statement. Individual operations — a specific attention pattern, a particular quantization format, a given batch shape — may differ more. The Chapter 23 methodology (measure at the right boundary, control variables, report percentiles) is the tool for answering platform-specific performance questions.

Memory capacity changes the design space. The MI300X’s 192 GB of HBM (and 288 GB on MI355X) changes which models fit on a single chip and how much KV state can be resident. The memory-budget worksheet from Appendix B applies unchanged — fill it in with the target platform’s capacity and bandwidth numbers. A model that requires tensor parallelism on 80 GB H100s may fit on a single MI300X, eliminating communication overhead entirely. Conversely, a platform with less memory per chip (32 GB on TPU v5e or Trainium) may require parallelism for models that fit on a single H100.

Compilation constraints are architectural, not incidental. On TPU and Trainium, compilation is not an optimization — it is the execution model. This changes the deployment workflow: warm-up is compilation, not graph capture; shape changes may require recompilation; and the bucket strategy from Chapter 9 is not optional but mandatory. Plan for compilation time in the deployment pipeline and in the capacity model.

Framework support is the practical boundary. The most important question for a non-NVIDIA deployment is not whether a principle transfers (it does) but whether the framework you use has implemented the platform-specific layer. vLLM supports NVIDIA, AMD, TPU, Gaudi, and Trainium backends. SGLang supports NVIDIA, AMD, and TPU (via its JAX backend). Check the framework’s backend maturity for your target platform — the presence of a backend does not guarantee feature parity with the NVIDIA path.

Appendix C. Reproducible Benchmark Cookbook

This appendix provides a compact format for the experiments used in the book. It is intentionally engine-neutral.

Benchmark card

Create one card per result.

decision: "Does configuration A improve TTFT/ITL-qualified goodput?"
hypothesis: "Smaller prefill chunks reduce decode stalls at a throughput cost."
date: "YYYY-MM-DD"

model:
  identifier: "MODEL_ID"
  revision: "MODEL_REVISION"
  tokenizer_revision: "TOKENIZER_REVISION"
  precision: "bf16"
  quantization: null

software:
  engine: "ENGINE_NAME"
  commit: "GIT_SHA"
  container_digest: "sha256:..."
  driver: "DRIVER_VERSION"
  accelerator_runtime: "RUNTIME_VERSION"

hardware:
  accelerator: "DEVICE_MODEL"
  count: 8
  cpu: "CPU_MODEL"
  host_memory_gib: 1024
  topology_file: "artifacts/topology.txt"

execution:
  tensor_parallel: 8
  pipeline_parallel: 1
  data_parallel: 1
  expert_parallel: 1
  graph_mode: "describe exact mode"
  attention_backend: "BACKEND"

workload:
  trace: "traces/workload.jsonl"
  arrival: "open-loop Poisson, 8 requests/second"
  warmup_requests: 200
  measured_requests: 5000
  cache_state: "empty at warm-up start; retained during measurement"
  seed: 1234

slo:
  ttft_p99_ms: 800
  itl_p99_ms: 80
  max_error_rate: 0.001
  quality_gate: "EVAL_NAME >= VALUE"

artifacts:
  command: "commands/run.sh"
  raw_results: "results/raw.jsonl"
  summary: "results/summary.json"
  traces: "results/traces/"

YAML is used here for readability. Store exact commands and raw data as files, not only in prose.

Workload trace schema

One JSON object per request is easy to stream and inspect:

{
  "request_id": "trace-000001",
  "arrival_ms": 0,
  "input_token_ids": [101, 202, 303],
  "max_output_tokens": 128,
  "priority": 0,
  "tenant_class": "interactive",
  "session_id": null,
  "media": [],
  "sampling": {"temperature": 0.0},
  "expected_schema": null
}

Production-derived traces should remove content and identifiers according to policy while preserving length, timing, prefix-sharing, and correlation needed by the experiment.

Experiment sequence

1. Verify model output and protocol behavior on a small golden set.
2. Record the environment and topology.
3. Run cold-start measurement if it is part of the decision.
4. Warm the intended compilation and cache paths.
5. Confirm that no unexpected compilation or fallback continues.
6. Run the workload at several offered-load points.
7. Capture detailed profiles only at representative regimes.
8. Repeat and retain every raw result, including failures.
9. Run quality and semantic-equivalence checks.
10. Write a conditional conclusion and its falsification boundary.

Required plots

For online generation, prefer:

• offered load versus SLO-qualified goodput;
• TTFT and ITL percentile curves versus load;
• queue time by stage;
• active batch and scheduled token distributions;
• cache matched tokens and transfer bytes;
• error, cancellation, and preemption rates;
• resource utilization by stage and rank;
• cost per qualifying request.

Do not truncate axes in a way that exaggerates small differences. Show uncertainty or repeated runs. Label cold, warm, and steady-state regions.

Fair comparison checklist

• Same model weights, tokenizer, template, precision, and quality target
• Same hardware allocation, power state, and topology
• Same request trace and arrival behavior
• Same context and output limits
• Same streaming and stop semantics
• Same cache starting condition
• Equivalent warm-up and compilation treatment
• Tuning policy described for every system
• Errors and timeouts retained in the denominator
• Raw commands and engine-specific configuration published

If equivalent semantics cannot be achieved, report the difference and avoid a single winner label.

Result statement template

Under [workload] on [hardware], using [model and revisions], configuration A
changed [primary metric] from X to Y while satisfying [quality and SLO gates].
The observed mechanism was [evidence from timeline/counters]. The result did
not hold under [boundary condition]. Raw artifacts are at [path].

Appendix D. Deployment Patterns and Decision Checklists

These patterns are starting points. Each diagram omits management components such as image registries, secret stores, and deployment controllers.

Pattern 1: One device

client -> API and engine -> one accelerator

Use this when the model and required state fit and one device meets the SLO. It has the fewest failure and communication paths. Add replicas before adding model parallelism when independent capacity is the goal.

Watch CPU preprocessing, memory headroom, and the difference between cold and steady behavior.

Pattern 2: Replicated single-node workers

                 +-> replica A (one or more local devices)
client -> router +-> replica B
                 +-> replica C

Use this for horizontal capacity and failure isolation. Choose routing based on load, session affinity, adapters, and cache locality. Keep replicas in independent failure domains where possible.

Watch fragmented caches, synchronized cold starts, and global overload.

Pattern 3: Multi-node model-parallel replica

router -> replica
          +-> node 0: ranks 0..7
          +-> node 1: ranks 8..15

Use this when model weights or state do not fit in one node. Map frequent tensor groups inside fast fabrics and use pipeline or expert boundaries deliberately across nodes.

Watch collective stragglers, pipeline bubbles, membership failure, and rank-to- topology mapping.

Pattern 4: Expert-parallel MoE service

requests -> attention/data-parallel groups
                     |
             expert dispatch fabric
            /       |        |       \
       expert ranks and optional replicas

Use this when experts dominate model size and conditional compute justifies distributed ownership. Select prefill- and decode-appropriate communication. Collect router traces and plan expert placement or replication.

Watch hot experts, network rails, grouped-GEMM shapes, and rebalancing safety.

Pattern 5: Prefill/decode disaggregation

                 +-> prefill pool -- KV transfer --+
client -> router |                              decode pool -> stream
                 +-> colocated pool (optional) ----+

Use this when phase interference or phase-specific scaling limits goodput. Retain a colocated path for requests whose transfer would not pay off if the router can estimate the choice reliably.

Watch coupled queues, transfer failures, pool ratios, and state accumulation between stages.

Pattern 6: Encoder/prefill/decode

media -> encoder pool -> feature transfer -> prefill pool
                                              |
                                           KV transfer
                                              |
                                              v
                                         decode pool

Use this for encoder-heavy multimodal traffic with independent batching or reuse. Cache media processing and encoder outputs at the appropriate trust boundary.

Watch feature identity, dynamic media shapes, two transfer boundaries, and first-output attribution.

Pattern 7: Hierarchical cache

GPU cache <-> host cache <-> local storage <-> distributed cache
    ^                                             |
    +----------- router and directory ------------+

Use this for expensive, reusable prefixes or session state that should survive one GPU. Separate directory metadata from bulk data transfer and make stale locations safe.

Watch promotion traffic, write policy, cross-tenant isolation, and cache-aware hotspots.

Pattern 8: Post-training loop

prompt source -> rollout pool -> rewards -> trainer pool
                    ^                         |
                    +---- weight transfer ----+

Use this when an inference engine generates online training data. Decide whether pools are colocated, alternating, or asynchronous. Version every trajectory and invalidate state after weight changes.

Watch long-tail groups, stale policy data, peak memory during updates, and mixed-rank failure.

Selecting a pattern

Begin with the simplest pattern that fits the model and SLO. Add a boundary only when it provides measurable value through independent scaling, state reuse, failure isolation, or hardware specialization. Every new boundary adds a queue, a protocol, a failure mode, and an observability requirement.

Decision checklists

This appendix collects the deployment decisions that practitioners face most often and compresses each into a structured checklist: what inputs you need, what procedure to follow, and where in the book the reasoning lives. Every checklist uses Atlas as its running example (70B dense decoder, BF16, 140 GB weights, 320 KiB KV per token, TP4 on 4x 80 GB GPUs) so the numbers are concrete, but the procedures generalize to any model.

1. Parallelism Configuration

Decision: How many GPUs, and what parallelism strategy?

Inputs: weight size in bytes, per-GPU memory, GPUs per node, model architecture (dense vs. MoE).

Procedure:

Does the model fit on one GPU?
  Test: weight_bytes < 0.8 x GPU_memory

  YES --> No parallelism needed.
          If tight (>0.7x), consider weight-only quantization
          to leave room for KV cache and activations.

  NO  --> Does it fit on one node with tensor parallelism?
          Test: weight_bytes / N_gpus_per_node < 0.8 x per_GPU_memory

          YES --> Use TP = N_gpus within the node.
                  Requires NVLink or equivalent between all TP ranks.

          NO  --> Two options:
                  (a) Add PP across nodes, TP within each node.
                      Budget for pipeline bubbles (~(PP-1)/micro_batches).
                  (b) Quantize to reduce weight size and retry the fit.

Atlas: 140 GB / 1 GPU = 140 > 64 (0.8 x 80). Does not fit on one GPU. 140 / 4 = 35 < 64. Fits with TP4 on one node.

MoE models: EP typically spans nodes (experts tolerate higher-latency interconnect). TP stays within each node. See Ch. 14 for EP/TP interaction.

Reference: Ch. 13, Ch. 10, Ch. 14

2. KV Cache Sizing

Decision: How much memory to reserve for the KV cache, and how many concurrent sequences you can serve.

Inputs: KV bytes per token (= 2 x n_layers x n_kv_heads x d_head x dtype_bytes), max context length, target concurrent sequences, available GPU memory after weights/activations/graphs/safety margin.

Procedure:

Step 1: Compute total KV demand.
  KV_total = KV_per_token x max_context x max_concurrent_sequences

Step 2: Compute available memory.
  available = GPU_memory - weight_shard - activation_overhead
              - graph_pools - safety_margin (5-10% of GPU memory)

Step 3: Does KV_total <= available?
  YES --> Proceed. Monitor utilization in production (checklist 10).
  NO  --> Reduce, in order of preference:
          1. max_concurrent_sequences (simplest, direct control)
          2. max_context (if workload permits)
          3. Quantize KV cache (FP8/INT8 halves KV memory)
          4. Add more GPUs

Atlas: 327,680 bytes/token x 4,096 ctx x 64 seqs = 85.9 GB. Per-GPU available: 80 - 35 - 3 - 2 - 4 = 36 GB, x4 GPUs = 144 GB. Fits. At 128K context the same calculation yields 2.75 TB – must reduce concurrency, context, or quantize KV.

Reference: Ch. 7

3. Chunked Prefill Configuration

Decision: What chunk size to use for chunked prefill.

Inputs: decode step time at target batch size, prefill cost per token, ITL target from your SLO.

Procedure:

Step 1: Start with a default chunk size of 512 tokens.

Step 2: Check the latency constraint.
  decode_step_time + prefill_chunk_time <= ITL_target
  max_chunk <= (ITL_target - decode_step_time) / time_per_prefill_token

Step 3: Check prefill efficiency.
  Very small chunks waste throughput due to per-step overhead.
  Profile prefill tokens/sec vs. chunk size; the knee is typically
  around 256-1024 tokens.

Step 4: Pick the largest chunk size at or above the efficiency knee
  that still satisfies the latency constraint.

Atlas: decode ~45 ms at batch 64, prefill ~0.035 ms/token, ITL target 150 ms. Max chunk: (150 - 45) / 0.035 = 3,000. Efficiency knee at ~512. Any value in 512-3,000 works; start at 512, increase if prefill throughput matters more than tight ITL control.

Caution: As the decode batch grows, decode_step_time rises and the budget for prefill chunks shrinks. Re-profile when you change concurrency.

Reference: Ch. 6

4. When to Enable Prefix Caching

Decision: Should you enable prefix caching (automatic prompt caching / RadixAttention)?

Inputs: workload prefix patterns, memory headroom (checklist 2).

Procedure:

Is there significant prefix reuse in your workload?

  System prompts shared across requests?  --> Strong benefit.
  Multi-turn conversations?               --> Yes, benefit grows with turns.
  RAG with repeated retrieval templates?  --> Template portion is cacheable.
  Unique prompts, no shared structure?    --> Low benefit; disable to save
                                              memory and hash overhead.

If enabled, verify in production:
  - Cache hit rate (vllm:cache_hit_rate or equivalent).
    Below 10-15% means the cache is not paying for itself.
  - Watch for eviction thrashing (high churn = wrong block size
    or too little memory allocated to cache).

Memory cost: Prefix caching holds KV blocks that might otherwise be freed. If KV cache is near capacity (checklist 2), enabling prefix caching can reduce max concurrency. Quantized KV caches help here.

Reference: Ch. 7, Ch. 16

5. When to Disaggregate Prefill and Decode

Decision: Should prefill and decode run on separate GPU pools?

Inputs: ITL distribution, prefill length distribution, KV transfer bandwidth, queue time statistics.

Procedure:

Step 1: Is prefill interfering with decode latency?
  Check: do ITL p99 spikes correlate with long-prompt arrivals?
  NO  --> Chunked prefill (checklist 3) is probably sufficient.
  YES --> Continue.

Step 2: Is the workload skewed?
  Long prompts, short outputs  --> prefill-heavy; dedicated pool helps.
  Short prompts, long outputs  --> decode-heavy; dedicated pool helps.
  Balanced                     --> less benefit; chunked prefill may suffice.

Step 3: Is the transfer cost acceptable?
  transfer_time = KV_size_for_prompt / link_bandwidth
  If transfer_time > queue_time_saved, disaggregation hurts.

Step 4: Do you have enough GPUs to staff two pools without
  creating new bottlenecks?

Atlas: A 4,096-token prompt = 1.28 GB KV. Over 25 GB/s link: 51 ms transfer. Worthwhile if it avoids queuing behind a 2-second prefill. For short prompts (256 tokens, 80 MB KV), the queuing delay is also short and disaggregation adds overhead for little gain.

Reference: Ch. 15

6. Quantization Selection

Decision: Which quantization method to use.

Inputs: BF16 quality baseline on your task, target GPU memory budget, hardware generation, kernel availability.

Procedure:

Step 1: Do you need to quantize?
  Model fits comfortably in BF16 with adequate KV budget --> skip.

Step 2: Try FP8 first.
  - Minimal quality loss (<0.5% on most tasks).
  - H100, MI300X, and newer. Often a single framework flag.
  - Halves weight memory vs. BF16.

Step 3: If FP8 unavailable or insufficient, try GPTQ or AWQ.
  - Weight-only INT4/INT8. Requires offline calibration.
  - Quality loss is task-dependent.

Step 4: Lower precision (INT4 weight + INT4 KV, GGUF).
  - Significant quality risk. Only for resource-constrained cases.

At every step:
  - Measure quality BEFORE and AFTER on YOUR task.
  - Verify optimized kernels exist for your hardware and shapes.
    Missing kernels cause silent fallback to slower paths.

Pitfall: A model that loses 1% on MMLU might lose 5% on your domain task. Generic benchmarks are not sufficient; always measure on your workload.

Reference: Ch. 10

7. Speculative Decoding: When It Helps

Decision: Should you enable speculative decoding?

Inputs: output length distribution, available GPU memory after weights and KV (checklists 1-2), draft model availability, grammar/constraint usage.

Procedure:

Step 1: Is the workload output-heavy?
  Short input, long output --> more opportunity for speedup.
  Long input, short output --> most time in prefill; less impact.

Step 2: Can you achieve a high acceptance rate?
  > 70%   --> likely beneficial.
  50-70%  --> marginal; profile carefully.
  < 50%   --> speculation wastes compute; disable.
  Measure on your actual workload, not generic text.

Step 3: Does the draft model fit in remaining memory?
  Its weights + KV must not evict target model KV capacity.
  If tight: use a very small draft (1-2B), Medusa heads, or EAGLE.

Step 4: Structured output or grammar constraints?
  Verify the implementation propagates grammar state to the draft
  model. Some do not, causing low acceptance on constrained output.

Rule of thumb: Helps most when memory-bound (GPU underutilized during decode) and the draft model predicts the target well. At high batch sizes where you are compute-bound, the extra computation may not pay off.

Reference: Ch. 11

8. Routing Policy Selection

Decision: How to distribute requests across replicas.

Inputs: replica count, prefix reuse pattern (checklist 4), adapter usage, telemetry refresh interval.

Procedure:

Single replica?
  --> No routing decision. Skip.

Multiple replicas, no prefix reuse, no adapters?
  --> Least-connections (uniform request cost) or
      least-estimated-work (variable cost; estimate from
      prompt length + expected output length).

Multiple replicas with prefix reuse?
  --> Hybrid cost score balancing:
      (a) Queue depth / estimated wait at each replica.
      (b) Prefix cache hit potential at each replica.
      A cache hit saving 500 ms of prefill is worth routing to
      a slightly longer queue.

Multiple replicas with adapters (LoRA)?
  --> Add adapter-locality term. Prefer replicas with the adapter
      already loaded; otherwise prefer most idle adapter slots.

Always:
  - Add uncertainty penalty proportional to telemetry staleness.
  - Implement fallback: retry on next-best if chosen replica rejects.

Reference: Ch. 17

9. Autoscaling Configuration

Decision: How to configure autoscaling for inference.

Inputs: end-to-end startup time, traffic pattern, latency SLOs, cost constraints.

Procedure:

Step 1: Measure startup time end-to-end.
  image_pull + weight_load + graph_compilation + warmup + health_check
  For Atlas: expect 3-8 minutes. This is your minimum reaction time.

Step 2: Choose the scale-out signal.
  DO NOT use GPU utilization -- it is a trailing indicator.
  USE: queue age, TTFT trend (rising p50/p95), or
       pending requests / available KV slots.
  Require N consecutive intervals above threshold to trigger.

Step 3: Size the warm pool.
  Must absorb spikes shorter than startup time.
  warm_pool >= peak_spike_requests / per_replica_throughput

Step 4: Configure scale-down with hysteresis.
  Scale out at queue_age > 2s.
  Scale in  at queue_age < 0.5s for 10+ minutes.
  The gap must be wide enough to prevent oscillation.

Step 5: Set minimum replica count.
  Never scale to zero unless you tolerate cold-start latency.

Pitfall: Tight scale-down hysteresis causes flapping. Each cycle wastes the full startup time and may spike latency. When in doubt, scale down slower.

Reference: Ch. 17, Ch. 24

10. “Is My Deployment Healthy?” Checklist

Decision: Is the deployment operating within acceptable bounds?

Run after deployment, periodically, and on any alert.

+----+------------------------------------+-------------+------------------+
| #  | Metric                             | Threshold   | Action if bad    |
+----+------------------------------------+-------------+------------------+
| 1  | Queue age                          | < 50% of   | Scale out or     |
|    |                                    | TTFT SLO    | reduce traffic   |
+----+------------------------------------+-------------+------------------+
| 2  | KV cache utilization               | < 85%       | Reduce concurr.  |
|    |                                    |             | or add capacity  |
+----+------------------------------------+-------------+------------------+
| 3  | Preemptions (last hour)            | 0           | KV undersized;   |
|    |                                    |             | see checklist 2  |
+----+------------------------------------+-------------+------------------+
| 4  | Graph/kernel fallback rate         | 0%          | Add missing      |
|    |                                    |             | shapes to warmup |
+----+------------------------------------+-------------+------------------+
| 5  | p99 TTFT                           | < SLO       | Check prefill    |
|    |                                    |             | sched, queue,    |
|    |                                    |             | prefix caching   |
+----+------------------------------------+-------------+------------------+
| 6  | p99 ITL                            | < SLO       | Check batch size,|
|    |                                    |             | chunked prefill  |
+----+------------------------------------+-------------+------------------+
| 7  | Error rate                         | < budget    | Investigate OOM, |
|    |                                    | burn rate   | timeout, upstream|
+----+------------------------------------+-------------+------------------+
| 8  | TP rank step time variance         | < 5%        | Straggler: check |
|    | (max - min across ranks)           |             | thermal, bad GPU |
+----+------------------------------------+-------------+------------------+

Key points: Preemptions mean the scheduler evicted a running request’s KV to make room – the evicted request recomputes from scratch, wasting GPU time. Graph fallback means a shape was not compiled and fell back to eager mode (2-5x slower). TP rank variance above 5% means the slowest GPU sets the pace for all ranks via the all-reduce barrier.

Reference: Ch. 17, Ch. 24

Decision Dependencies

Some decisions feed into others. Work through them in this order:

  Parallelism (1) --> KV cache sizing (2) --> Chunked prefill (3)
                            |                        |
                            v                        v
                      Prefix caching (4)      Disaggregate P/D (5)
                            |
                            v
                      Routing policy (8)

  Quantization (6)     <-- feeds back into (1) and (2) if memory is tight
  Spec. decoding (7)   <-- depends on memory headroom from (2)
  Autoscaling (9)      <-- uses latency targets affected by (3), (5), (6)
  Health check (10)    <-- validates all of the above in production

Start with parallelism, then KV sizing, then work through the rest. Revisit earlier decisions when a later checklist surfaces a constraint you missed.

Chapter Reference Summary

Appendix E. Glossary

Definitions in this glossary describe concepts as used in the book. Frameworks may use the same term differently.

Acceptance rate — The fraction of speculative proposal tokens accepted by the target model. It does not include draft or verification cost and is not a speedup by itself.

Adapter — A small weight delta applied to a base model’s parameters at inference time. LoRA is the most common form; the adapter modifies a subset of layers through low-rank matrix additions without changing the base weights.

Admission control — The decision to accept, delay, redirect, or reject new work based on capacity and service objectives.

All-gather — A collective in which every rank receives the shards held by all ranks.

All-reduce — A collective that reduces values across ranks and returns the result to every rank.

All-to-all — A collective in which each rank sends a distinct portion of data to every other rank. Expert dispatch and combine often use this pattern.

Arithmetic intensity — Operations performed per byte moved across a chosen memory boundary.

Attention backend — A selected implementation of attention for a device, model pattern, dtype, cache layout, and execution mode.

Batch invariance — A numerical contract under which a request’s output is unchanged by the other requests with which it is batched.

Block table — A mapping from logical sequence blocks to physical cache blocks.

Cache-aware routing — Request placement that considers reusable state as well as load and other constraints.

Chunked prefill — Processing a long prompt across several engine steps so other work, especially decode, can advance between chunks.

Closed-loop load generation — A workload in which a client waits for a response before issuing more work. Offered load falls when the server slows.

Collective — A communication operation involving a group of ranks, such as all-reduce, all-gather, reduce-scatter, or all-to-all.

Compilation artifact — Generated code, a graph, tuning result, or other reusable output tied to an execution environment and configuration.

Constrained decoding — Generation that masks tokens according to a grammar, schema, regular expression, or other allowed-output state.

Context parallelism (CP) — Partitioning sequence positions or context work across ranks. The exact algorithm must be stated.

Continuous batching — Changing batch membership between engine steps so completed sequences leave and waiting sequences enter.

Control plane — Components that place, route, scale, and recover work across engines rather than executing the current model step.

Copy-on-write — Sharing immutable state until a writer needs to modify it, at which point a private copy is created.

CUDA Graph — A recorded GPU operation graph that can be instantiated and replayed with reduced launch overhead.

Data parallelism (DP) — Replicating a model or model component so ranks can process independent request work. In MoE deployments, attention DP may compose with shared expert parallelism.

Data plane — The request-critical path that schedules and executes current work, manages its state, and produces output.

Decode — The autoregressive phase that adds output positions, usually one per active sequence per ordinary engine step.

Decode-context parallelism (DCP) — Partitioning decode context or KV state by sequence position and combining partial attention results.

Degradation ladder — A structured sequence of service reductions applied under overload, ordered from least to most visible impact on the user.

Disaggregation — Placing model stages, such as prefill and decode, in separate worker pools with explicit intermediate transfer.

E/P/D — Encoder/prefill/decode disaggregation.

End-to-end latency (E2E) — Time from request arrival at the measured service boundary to final response completion.

Engine step — One scheduler decision and its corresponding model execution and output update. Some chapters use “model step” for the same cycle.

Eviction — Removing reusable cached state, such as prefix blocks, to free capacity under a retention policy. Distinct from preemption, which removes running work, and from swapping, which moves a suspended request’s state.

Execution identity — The combination of model version, processor configuration, and cache-relevant parameters that determines whether cached state is valid for a given request.

Expert parallelism (EP) — Distributing different MoE experts across ranks and routing token representations to their owners.

Expert-parallel load balancing (EPLB) — Changing expert placement or replication based on observed routing load.

Generation fence — A monotonic version counter on a session or request that invalidates late-arriving work from a superseded generation, preventing stale results from reaching the output stream.

Goodput — Completed work per time that satisfies a stated latency, correctness, quality, and error contract.

Graph bucket — A captured or compiled execution shape selected to cover a range of runtime batches, often with padding.

Grouped GEMM — Execution of several matrix multiplications, often with different shapes, through one coordinated operation. Common in MoE layers.

HBM — High-bandwidth memory attached to an accelerator.

Hierarchical cache — A cache that places state across tiers such as GPU, host memory, local storage, and remote storage.

Inter-token latency (ITL) — The time between consecutive visible output tokens or stream events.

JIT compilation — Generating or specializing executable code at runtime.

KV cache — Persistent attention keys and values created from earlier token positions. The term is sometimes used loosely for other model-specific sequence state.

KV connector — An engine interface or implementation that moves or stores KV state outside its local cache manager.

Latency percentile — A value below which a stated percentage of observations falls within a defined population and window.

LoRA — Low-rank adaptation. A parameter-efficient fine-tuning method that adds low-rank matrix pairs to selected layers. At inference time, the adapter weights are merged or applied alongside the base model’s forward pass.

Management plane — Deployment and policy systems that change the service’s configuration, software, model, or capacity.

Membership epoch — A versioned snapshot of which workers belong to a distributed group. Stale membership information self-invalidates when the epoch advances, preventing routing to departed or unhealthy members.

Model runner — The engine component that prepares device tensors and invokes model code, kernels, graphs, and collectives for a scheduled step.

MoE — Mixture of experts, a model layer that routes each token to a subset of expert networks.

Multi-head latent attention (MLA) — An attention architecture that stores and operates on compressed latent representations rather than a conventional full KV layout.

NUMA — Non-uniform memory access, in which CPU memory access cost depends on the socket or node that owns the memory.

Open-loop load generation — A workload that sends requests according to an external arrival process regardless of current server latency.

P/D — Prefill/decode disaggregation.

Paged attention — Attention over KV state stored in noncontiguous physical blocks addressed through logical mappings.

Pipeline parallelism (PP) — Assigning ranges of model layers or stages to different ranks and sending activations between them.

Prefix cache — Retained model state for a reusable beginning of an input.

Prefill — Processing input tokens or positions to produce the first output and persistent state for later decode.

Preemption — Removing a running request from active execution to free capacity, with later recomputation or state restoration.

Prompt cache — A broad term for reusable results derived from prompts. State the representation: tokens, processed media, encoder features, or KV blocks.

Radix cache — A prefix cache indexed by a radix tree over token or other discrete input sequences.

Rank — One member of a distributed process group, identified by an integer within that group.

RDMA — Remote direct memory access, a family of mechanisms for moving data between registered memory regions with reduced CPU involvement.

Reduce-scatter — A collective that reduces values and leaves a different shard of the result on each rank.

Release identity — The pinned combination of model weights, tokenizer, chat template, grammar backend, runtime image, and configuration that uniquely identifies a deployment version. Changes to any component require re-running conformance and benchmark suites.

Request goodput — Requests completed within a specified service and quality contract per unit time.

Roofline model — A performance bound that compares peak compute with memory bandwidth times arithmetic intensity.

Router — A control-plane component that assigns new requests or stages to engines or worker pools.

Scheduler — The data-plane component that chooses which admitted work advances in the next engine step.

Sequence parallelism (SP) — A family of methods that shard sequence-related work or tensors. Define the specific data layout and communication when using the term.

Service-level objective (SLO) — A measurable target for latency, availability, correctness, quality, or another service property.

Session affinity — Routing related turns or events to the same worker to preserve local state.

Sleep level — A graduated resource-release mode used when an inference worker yields capacity to a trainer or autoscaler. Deeper levels release more resources (KV state, then weights from device memory) and cost more to resume.

Speculative decoding — Proposing future tokens with a cheaper method and verifying them with the target model so one target step may advance several tokens.

State-space model (SSM) — A sequence model that maintains recurrent state instead of or alongside token-indexed attention state.

Structured output — Output constrained or interpreted according to a schema, grammar, tool protocol, or parser contract.

Swapping — Freeing capacity by moving a suspended request’s state to host memory so it can be restored later without recomputation; the alternative to recomputing after preemption. The state occupies host capacity for the whole suspension.

Tensor parallelism (TP) — Sharding matrices and operations within model layers across ranks, usually with frequent collectives.

Time per output token (TPOT) — Average post-first-token generation time per additional output token. It can hide individual stream stalls.

Time to first token (TTFT) — Time from request arrival to the first visible output token or equivalent event.

Token budget — A scheduler limit on total token positions processed in one engine step.

Topology — The physical or logical arrangement of devices, links, ranks, model stages, and state owners.

Warm-up — Deliberate execution before measured or user traffic to load weights, allocate memory, compile kernels, tune implementations, and capture graphs.

Weight transfer — A protocol that moves updated model parameters from a trainer or source to inference ranks.

Write-back cache — A policy that copies modified or newly created state to a lower tier later, often near eviction.

Write-through cache — A policy that writes state to a lower tier as it is created or admitted to the upper tier.

Field names

The book teaches concepts under system-neutral names so the reasoning outlives any project’s vocabulary. This map connects them to the canonical names used in papers, engines, and interview conversations. Names evolve; check the source ledger (Appendix F) for the pinned revisions behind each.

Appendix F. Source and Reproducibility Ledger

This edition distinguishes principles, implementation observations, and measurements. A repository path proves that code exists at a snapshot; it does not prove production readiness or performance on every platform.

Edition snapshot

The supplied book informed the coverage audit and standards for approachability. Its prose, diagrams, examples, analogies, and chapter sequence were not reused.

Primary systems papers

• Ashish Vaswani et al., Attention Is All You Need, 2017.
• Woosuk Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, 2023.
• Lianmin Zheng et al., SGLang: Efficient Execution of Structured Language Model Programs, 2023.
• Gyeong-In Yu et al., Orca: A Distributed Serving System for Transformer-Based Generative Models, OSDI 2022.
• Amey Agrawal et al., Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve, 2024.
• Yinmin Zhong et al., DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving, 2024.
• Pratyush Patel et al., Splitwise: Efficient Generative LLM Inference Using Phase Splitting, 2023.
• Ruoyu Qin et al., Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving, 2024.
• Lijie Liu et al., Preble: Efficient Distributed Prompt Scheduling for LLM Serving, 2024.

Kernels, execution, and decoding

• Tri Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, 2022.
• NVIDIA, CUDA Programming Guide: CUDA Graphs.
• PyTorch, torch.compile reference.
• Yaniv Leviathan, Matan Kalman, and Yossi Matias, Fast Inference from Transformers via Speculative Decoding, 2022.
• Guangxuan Xiao et al., SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, 2022.

Adapter serving

• Ying Sheng et al., S-LoRA: Serving Thousands of Concurrent LoRA Adapters, 2023.
• Chen Liang et al., Punica: Multi-Tenant LoRA Serving, 2023.
• Edward J. Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, 2021.

Parallel and MoE systems

• Mohammad Shoeybi et al., Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism, 2019.
• Deepak Narayanan et al., Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM, 2021.
• DeepSeek, DeepEP, official expert-parallel communication implementation.

Media and post-training

• Xinyin Ma, Gongfan Fang, and Xinchao Wang, DeepCache: Accelerating Diffusion Models for Free, 2023.
• Feng Liu et al., Timestep Embedding Tells: It’s Time to Cache for Video Diffusion Model, 2024.
• Wei Fu et al., AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning, 2025.
• Shuo-yiin Chang et al., Joint Endpointing and Decoding with End-to-End Models, 2019.
• Loïc Barrault et al., Seamless: Multilingual Expressive and Streaming Speech Translation, 2023.

Standards and operating references

• NVIDIA, DCGM: Topology and NVLink, for distinguishing topology inventory, link state, and measured traffic.
• MLCommons, MLPerf Inference documentation, for scenario, accuracy, and run-rule discipline.
• OpenTelemetry, Semantic conventions, for consistent trace, metric, log, and resource naming.
• NIST, AI Risk Management Framework and Generative AI Profile.
• OWASP, Top 10 for LLM Applications.

Official implementation documentation

• vLLM documentation, including architecture, cache, parallelism, disaggregation, compilation, multimodal, and training integration guides.
• vLLM architecture overview.
• vLLM batch invariance and pooling models.
• SGLang documentation, including attention backends, distributed serving, observability, post-training, and diffusion.
• SGLang deterministic inference, reasoning parsers, embeddings, and prefill/decode disaggregation.
• SGLang Diffusion.

Documentation can describe a release different from the pinned source snapshot. When the two conflict, the manuscript either describes the pinned code or marks the behavior as release-dependent.

Publication tooling

Block diagrams use a purpose-built SVG renderer, with Dagre 3.1.1 providing directed-graph coordinates. Dagre and its license are vendored into the book (assets/vendor/), so diagrams render offline and are not affected by CDN changes. The book controls block dimensions, label wrapping, connectors, decision shapes, and responsive reflow; the diagram definitions remain readable as text if the client-side renderer cannot load. Body, interface, and code typefaces (Literata, Inter, JetBrains Mono) are also vendored as subsets. Diagram definitions remain readable as text if the client-side renderer cannot load.

Implementation map by chapter

Reproducibility status

This manuscript explains how to design experiments but does not claim new performance results. Numeric results cited from papers remain the authors’ results under their published setups. Future editions should attach original benchmark cards, traces, commands, and raw data here, with each claim marked:

• proposed — experiment designed but not run;
• reproduced — run with public artifacts;
• reviewed — independently checked;
• superseded — retained for history but replaced by newer evidence.

Appendix G. Worked Solutions

The exercises use a continuing fictional service called Atlas, an enterprise research assistant. Atlas serves a dense decoder with approximately 70 billion parameters. The baseline uses BF16 weights, 80 transformer layers, 8 KV heads, and head dimension 128. Traffic is 70 percent short interactive questions, 20 percent document questions, and 10 percent long research jobs. Many requests share tenant instructions or uploaded documents.

These are worked engineering answers, not universal configurations. Where an exercise would normally require measurements, the solution states assumptions and shows the decision process instead of inventing results.

1. Request trace

Assume a user sends a 6,000-token document question and asks for at most 300 output tokens. A defensible trace is:

client
  -> edge queue
  -> API validation and tenant lookup
  -> tokenizer queue
  -> cache-aware router
  -> engine admission queue
  -> prefix lookup and KV allocation
  -> prefill steps
  -> decode steps
  -> detokenizer and stream buffer
  -> client

The request record is owned by the API until accepted by the engine, then by the engine until final cleanup. The block manager owns KV allocation. The router owns only a possibly stale location hint; it does not own the cached blocks. Cancellation can arrive while queued, during prefill, during decode, or after a token has entered the output buffer. In every case the engine stops future scheduling, marks in-flight work discardable, and releases blocks only after the last GPU user finishes.

The least visible delay in this example is the cache-aware routing decision. The chosen replica saves 4,000 prompt tokens but has 450 ms of queued prefill; an idle replica could recompute them in 240 ms. Locality is therefore a loss. The trace needs both queue_age_at_assignment and matched_prefix_tokens to make that diagnosis possible.

A complete answer names the owner and queue at every boundary. A box-and-arrow diagram without those labels does not reveal cancellation safety or latency.

2. Workload traces and goodput

Construct 100 requests with 100,000 total input tokens and 20,000 total output tokens in each trace:

Use an open-loop arrival rate of 8 requests/s. Define a qualifying interactive request as successful, TTFT at most 600 ms, no ITL above 150 ms, and valid output. If 100 requests arrive over 12.5 seconds, 96 finish, and only 81 meet all conditions, throughput is 96 / 12.5 = 7.68 requests/s while goodput is 81 / 12.5 = 6.48 requests/s.

The even trace should have the narrowest latency distribution. The bursty trace can have identical token throughput yet lower goodput because long prefills delay active decoders. The conversational trace should benefit from prefix reuse, but only if routing does not create a hot replica. A closed-loop rerun will probably look healthier because slow responses reduce the offered load; that is a property of the generator, not an engine improvement.

The correct report preserves each trace’s length and arrival correlations. It does not shuffle prompt lengths independently or average all traffic classes into one latency number.

3. Model topology inventory

For the Atlas dense decoder, BF16 parameter storage is approximately 70 billion × 2 bytes = 140 GB, before allocator and runtime overhead. Its KV state per token is:

2 × 80 layers × 8 KV heads × 128 dimensions × 2 bytes = 327,680 bytes

That is 320 KiB per token across the model, or about 2.44 GiB for an 8,000-token sequence. Tensor parallelism shards this state, but does not make the aggregate bytes disappear.

For a hypothetical 8 × 7B expert model with two experts active per token, total BF16 expert weights are roughly 112 GB plus shared attention and router weights. Per-token arithmetic uses two experts, while expert parallel serving adds dispatch and combine traffic. The topology inventory therefore includes both total resident bytes and active bytes per token.

For a hybrid vision-language model, list the vision encoder, projection, dense decoder, encoder outputs, and decoder KV state separately. A 2,048-token image feature sequence reused across five questions is a candidate for encoder-output caching or encoder disaggregation. Its transfer size, not the original JPEG size, governs that boundary.

The serving conclusion is not “MoE is cheaper” or “hybrid is slower.” The dense model is dominated by replicated or sharded weights and growing KV state. MoE adds conditional placement. The hybrid adds a separately batchable encoder and reusable feature state. Those facts determine which serving plans are legal.

4. Topology prediction

Assume two eight-GPU nodes. Each GPU has 80 GiB of device memory; links inside a node are fast, and the inter-node network is slower. Place one four-way tensor-parallel Atlas replica entirely inside each half-node. At 140 GB of BF16 weights, each rank holds roughly 35 GB before non-parameter overhead. The KV state is also sharded: an 8,000-token request holds about 625 MiB per rank.

The initial prediction is:

Do not stripe one TP group across both nodes unless measurement shows the inter-node collectives are acceptable. The arithmetic is unchanged, but each layer now depends on the slower fabric.

Profile counters should be chosen to falsify the prediction: achieved compute and HBM bandwidth for prefill; memory bandwidth, GPU gaps, and collective time for decode; payload bytes, staging copies, and concurrent-link throughput for the KV move. If decode shows low HBM traffic and large CPU gaps, the original prediction was incomplete—the host or launch path is the actual limit.

5. Control and data paths

For one Atlas request, the control path is:

submit -> validate -> admit -> schedule -> allocate -> execute -> finish

The data path is:

text -> token IDs -> model tensors -> KV blocks -> logits -> token IDs -> text

They meet at several synchronization points. Admission waits for an ownership decision so rejected work cannot allocate state. The model runner waits for the scheduler’s block table because attention must address the right physical pages. The sampler waits for logits because the next token is a true dependency. Cleanup waits for the completion event of in-flight GPU work so memory is not reused early.

Other waits are candidates for overlap. Tokenizing request B can overlap model execution for request A. Output processing for step n can overlap GPU work for step n + 1 if request-state updates are versioned. A cache write-back can be asynchronous if eviction and failure do not make the only valid copy disappear.

The answer should attach an invariant to every wait. “The CPU waits for the GPU” is description; “cleanup waits so no block can be reallocated while the GPU still holds its address” is an engineering explanation.

6. Scheduler simulation

Use a token budget of 16 per step and enough memory for 40 live token-block units. Four requests arrive:

With first-come-first-served and no chunking, A consumes the first two steps and B’s first token waits. With an 8-token prefill chunk, step 0 can schedule eight tokens of A and four of B. B enters decode earlier; later steps mix one decode token for B and C with bounded chunks of A or D.

A reasonable priority-with-aging score is:

score = base_priority + 0.05 × waiting_steps

Always reserve enough of the step budget for active decoders before scheduling prefill. Reject D if its predicted completion is already beyond its deadline or if admitting its state would force higher-priority C to preempt.

The simulator should use a measured step-time table such as (decode batch, prefill tokens) -> milliseconds. Counting every simulated step as equal would hide the interference being studied. Report TTFT and deadline-qualified goodput per class, not only the number of scheduled tokens.

7. KV-cache correctness matrix

Begin with a known 512-token prefix and change one identity dimension at a time:

Now cancel a writer after the GPU has produced a partial final block but before publication. The block remains private and pinned until the completion event; it is then discarded or published atomically according to the cancellation policy. A branch shares only sealed full blocks. Its partial tail is copied on write. Eviction removes the reusable index entry first, then frees storage only when the reference count reaches zero.

Three invariants make the test decisive: no lookup returns an unpublished block; no physical block is writable by two logical branches; and every terminal request eventually releases its references. Output equivalence should be checked against a cache-disabled run, not merely against another cached run.

8. Kernel evaluation

Suppose a new paged-attention kernel is 22 percent faster than the baseline for batch 32, context 4,096, head dimension 128. The isolated result is only level one.

At engine-step level, add metadata construction, block-table transfer, layout conversion, sampling, and synchronization. If the old step takes 5.0 ms, of which attention is 2.0 ms, a 22 percent attention improvement saves at most 0.44 ms before new overhead. A 0.3 ms conversion leaves the step only 2.8 percent faster: (5.0 - 0.14) / 5.0.

At workload level, assume the kernel requires 64-token pages instead of 16-token pages. Short prefixes now waste more cache tail space and exact reuse ends at coarser boundaries. Lower cache capacity can create preemption or recomputation that erases the 0.14 ms step gain.

The decision is therefore conditional: enable the kernel only for supported shapes or page layouts until end-to-end goodput improves. Correctness tests cover single-token decode, non-multiple context lengths, partially filled pages, empty sequences, mask boundaries, and tolerances against a trusted implementation. A fast result for one rectangular shape does not justify a global backend switch.

9. Compilation and graph buckets

Assume one minute of Atlas traffic produces decode batch sizes:

1: 8%, 2-4: 17%, 5-8: 31%, 9-16: 29%, 17-32: 15%

Candidate graph buckets are 1, 4, 8, 16, and 32. Dispatch rounds a batch up to the smallest compatible bucket. Batch 9 therefore uses bucket 16 and executes seven padded slots. The trace should record both the requested and replayed shape so padding is visible.

Compare three regimes after a separate cold-start measurement. If eager uses 1.1 ms CPU launch time and 4.0 ms GPU time at batch 8, graph replay that reduces CPU time to 0.2 ms but adds 0.15 ms padding/dispatch work improves the step from 5.1 to 4.35 ms. At batch 9 replaying bucket 16 might add enough padded GPU work to lose.

The response is not to disable graphs. Add a bucket near a frequent costly gap, allow eager fallback for rare shapes, and set a memory budget for captured artifacts. Report compilation time separately, plus artifact count, bucket hit rate, padding ratio, fallback rate, graph memory, and SLO-qualified goodput.

10. Quantization decision

Compare BF16, weight-only INT4, and FP8 weights plus FP8 KV state. Use the same Atlas request trace and product evaluation set.

Do not infer speed from the table. Measure representative prefill and decode shapes. If INT4 saves weight bandwidth but its kernel performs poorly at batch 1, it may increase interactive ITL. If FP8 KV doubles capacity, it may improve goodput by avoiding preemption even when one isolated attention call is unchanged.

Gate quality using product tasks, tool-call validity, long-context retrieval, rare languages, and log-probability drift. A sensible decision record might choose FP8 for the long-document tier, retain BF16 for sensitive evaluation, and reject INT4 until the target small-batch kernel improves. The answer names the binding service constraint; “FP8 won” is incomplete.

11. Speculation break-even

Let ordinary target decode cost 8 ms per token. A speculative step proposes four tokens, costs 3 ms to draft, and 9 ms to verify. If it accepts an average of 3.2 tokens, cost per accepted token is (3 + 9) / 3.2 = 3.75 ms, a strong win before system overhead.

On an unpredictable prompt, suppose only 1.3 tokens are accepted. Cost becomes 12 / 1.3 = 9.23 ms, slower than ordinary decode. At high concurrency, the draft model’s memory may also reduce target KV capacity, while verification uses larger shapes that contend with other requests.

The serving policy should estimate expected accepted tokens and compare:

draft cost + verify cost + capacity cost
        versus
ordinary cost × expected accepted tokens

Use a conservative threshold and turn speculation off for short remaining outputs, low historical acceptance, memory pressure, or graph-incompatible shapes. Verify output distribution equivalence for the algorithm in use; equal tokens under one random seed are useful debugging evidence but not the whole distributional contract.

12. Adapter routing simulation

The exercise asks for adapter-aware versus adapter-blind routing with Zipf-distributed popularity across 50 adapters.

Model the adapter popularity with a Zipf distribution (α = 1.1). The top 5 adapters receive roughly 45% of requests; the bottom 30 receive about 15% combined. With three replicas and round-robin routing, each replica must load all 50 adapters on demand — at 160 MiB per adapter and 3 ms host-to- device transfer, cold loads add 3 ms per miss and 50 × 160 MiB = 8 GiB of adapter state per replica in steady state.

With adapter-aware routing, partition adapters by popularity: the top 10 (hot set) replicate everywhere; the next 20 partition across two replicas each; the bottom 20 partition to one replica each. Each replica holds 10 + 10 + ~7 ≈ 27 adapters in steady state (4.3 GiB), and cold loads drop to only the tail adapters accessed for the first time. Monitor four quantities: cold load rate per replica, adapter memory per replica, max queue depth per replica, and p99 TTFT.

The key insight: the routing score must penalize queue depth enough that adapter affinity does not create hot-replica bottlenecks. When the busiest adapter attracts 12% of traffic, sending all of it to one replica creates a 3× load imbalance. The solution is a composite score with load weight exceeding affinity weight, as Chapter 12’s worked example demonstrates.

13. Two parallel plans

Use eight GPUs in one fast-link island for one Atlas replica.

Plan A: tensor parallel size 8. Every layer shards large matrix operations and performs layer-frequency collectives across all eight ranks. Weight and KV memory per rank are smallest, and there is no pipeline bubble, but decode pays the widest collective latency.

Plan B: pipeline size 2 and tensor parallel size 4. Two groups of four own half the layers. TP collectives stay within smaller groups; one activation tensor crosses the stage boundary. A single interactive request creates a pipeline bubble, while enough concurrent microbatches can fill both stages.

For a hidden width of 8,192 and BF16 activation, one token’s stage-boundary payload is about 16 KiB per sequence before batching. TP collective volume is implementation-dependent, so the answer must derive it from the chosen sharding algorithm rather than assert one universal formula.

Predict Plan A for low-concurrency prefill if its all-rank fabric is excellent and minimizing bubbles matters. Plan B may win for sustained throughput when four-rank collectives are materially cheaper and concurrency fills the pipeline. Measure prefill and decode separately; one winner is not required.

14. Expert trace and placement

Assume one MoE layer has eight experts on four ranks, two experts per rank. A decode step routes 64 tokens with counts:

E0 22, E1 14, E2 7, E3 6, E4 5, E5 4, E6 3, E7 3

With contiguous placement (E0,E1) on rank 0, that rank receives 36 expert assignments while rank 3 receives 6. The layer finishes at the hot rank. Moving E1 to rank 3 produces loads 25, 13, 9, and 17 if paired carefully. The maximum falls, although remote dispatch paths change.

If the activation width is 8,192 in BF16, one routed copy is 16 KiB. For top-2 routing, 64 tokens create 128 assignments, or roughly 2 MiB of activation payload before protocol overhead for dispatch and another combine transfer. The trace should count assignments, not unique input tokens.

An EPLB update is worthwhile only if reduced straggler time exceeds weight-copy and cache disturbance. Apply a generation number: new batches use the new placement only after every rank acknowledges it; old batches finish under the old map. Keep a fallback copy during the transition if memory allows.

15. Prefill/decode split

Assume measured service models:

prefill_ms(tokens) = 20 + 0.035 × tokens
decode step at target batch = 45 ms
KV transfer = 12 ms setup + bytes / 22 GiB/s

Atlas KV state is 320 KiB per token. A 6,000-token prompt creates about 1.83 GiB, so its idealized transfer takes roughly 85 ms plus 12 ms setup. The prefill itself is approximately 230 ms. Transfer is therefore a material stage, not a rounding error.

Disaggregation can still win if decode no longer waits behind long prefills and the two pools are independently saturated. Model three queues: prefill, transfer, and decode. Admission requires a predicted decode slot, not only an idle prefill worker.

Use conditional placement: keep short uncached prompts colocated; disaggregate long prompts when the expected reduction in decode interference exceeds transfer and extra queueing. On transfer failure, discard the unpublished destination state and either retry within the deadline or recompute. Report stage utilization because low end-to-end latency obtained with a mostly idle pool may be economically unacceptable.

16. Distributed prefix lifecycle

Give the prefix an identity derived from model version, tokenizer, adapter, tenant namespace, token sequence, position scheme, and state format. GPU A creates blocks privately, seals them after execution, and publishes metadata only after the data is readable.

Host backup takes a read reference and copies the sealed blocks. A successful checksum and generation match publish the host location. Remote metadata may advertise that location, but it is a hint: GPU B obtains a lease, revalidates identity, reserves destination blocks, transfers, verifies, and only then inserts them into its local index.

If B cancels mid-transfer, the destination blocks stay unpublished and are released after the copy completion or abort event. A metadata timeout causes a miss and recomputation, not indefinite request blocking. Invalidation removes new lookup visibility first; existing readers finish through references or leases; physical deletion follows when references reach zero.

For a 1-GiB prefix that saves 180 ms of prefill but needs 70 ms to load, the gross request saving is 110 ms. Include queueing and the opportunity cost of 1 GiB on the destination before deciding to promote it. A hit counter alone cannot express that value.

17. Cache-aware routing

Assume three replicas. R0 has a 4,000-token prefix but 300 ms of queued work; R1 is idle without the prefix; R2 has half the prefix and 100 ms queued. Recomputation costs 0.06 ms per missing prompt token.

Estimate completion contributions:

R2 wins this simplified decision. Cache-only routing would choose R0 and least- queue routing would choose R1; both ignore useful information. Add transfer cost, uncertainty, adapter availability, and deadline risk in a real score.

When one prefix becomes very hot, one cached replica becomes a queue hotspot. Replicate the prefix only when predicted avoided recomputation and queue relief justify its memory. Use hysteresis so placement does not oscillate with every small popularity change. The simulator should delay telemetry deliberately; perfect instantaneous queue knowledge would make the router unrealistically powerful.

18. Multimodal first-output path

For a repeated image question, suppose the first request has this trace:

TTFT is 465 ms. A processed-image cache saves only the 28 ms preprocessing stage. An encoder-output cache saves preprocessing, encoder queue, encoder, and possibly feature creation—183 ms here—at the cost of retaining a larger, model-version-specific tensor. Full language-prefix reuse may save still more but is invalid if the question tokens occur inside the reusable prefix boundary.

The second question must retain the same media identity, preprocessing configuration, model version, and feature layout. Compare output with a cache- disabled request. If disaggregating the encoder adds a 35 ms transfer instead of 12 ms but removes an 80 ms queue at the language worker, it improves TTFT; for uncached tiny images, the extra boundary may lose.

19. Diffusion timeline

Assume an image pipeline spends 18 ms in text encoding, 30 denoising steps at 24 ms each, 55 ms in latent decoding, and 22 ms postprocessing. Total service time is 18 + 720 + 55 + 22 = 815 ms; denoising is the clear target.

A cache that skips equivalent work in 10 steps saves at most 240 ms before lookup and correction cost. It must be evaluated with prompt classes and image quality, not only cache hit rate. At double spatial resolution, latent work can grow far faster than the text stage, so the bottleneck conclusion should be retested.

For disaggregation, text embeddings are small and easy to transfer, while the latent entering the decoder can be large. Separate a stage only if independent scaling, reuse, or batch compatibility offsets the transfer and queue. A good timeline records every denoising step, graph bucket, cache action, synchronization, and stage-boundary byte count.

Report paired blinded samples and a declared quality metric. A latency win that changes composition or temporal consistency is a different product setting, not a free acceleration.

20. Policy update transaction

Every rollout carries policy_version = 41. The trainer produces version 42 in staging storage with a manifest of tensors, shapes, dtypes, and checksums. Inference ranks stop admitting version-41 groups, finish or abandon them by policy, enter sleep or update state, and copy version 42 into inactive buffers.

Each rank validates the manifest and reports prepared(42). Only after all ranks prepare does the coordinator publish commit generation 42. Ranks swap buffers, invalidate version-dependent KV and graph artifacts, run a health forward pass, and report ready. New rollout admission then resumes with version 42.

If rank 3 fails halfway through copy, no commit is published. Prepared ranks retain version 41 as active and discard or retry their inactive buffers. The service never constructs one tensor-parallel group from mixed versions.

When the trainer is delayed, bound rollout work by both queue bytes and maximum policy lag. Stop admission before trajectories grow without bound. The trainer may accept completed version-41 groups if the algorithm allows that lag; the serving layer must not invent the rule. Log token IDs, masks, sampling state, and log-probability semantics so training can reproduce or deliberately trust the rollout calculation.

21. Ten-second conversation

One coherent timeline is:

0.0-2.4  user speech; ASR partials at 0.8, 1.5, 2.2
2.4-2.6  endpoint stabilization and final transcript
2.6-2.8  LLM routing and prefill
2.8-3.0  first text; tool call emitted
3.0-3.6  tool runs; assistant sends a brief holding phrase
3.6-3.8  final response begins; TTS buffers first audio
3.8-5.1  assistant audio plays
5.1      user interrupts; turn generation changes from 7 to 8
5.18     playback is silent
5.4      stale TTS chunk for generation 7 arrives and is discarded

The end-of-turn-to-first-audio budget is 1.4 seconds here: 200 ms endpointing, 200 ms language startup, 600 ms tool time overlapped with a holding phrase, and 200 ms TTS plus buffer, with remaining transport margin. The more critical interruption budget is 100 ms from new speech detection to silence.

Every event carries session ID, turn generation, monotonically increasing sequence number within its stream, and a deadline. Advancing the generation cancels future LLM scheduling and TTS, stops playback, and makes late generation- 7 events inert. Only text actually heard by the user is committed to the visible conversation history; generated-but-unplayed text is recorded as diagnostic state, not silently treated as spoken.

22. Protocol conformance

Define one golden request with a pinned tokenizer and chat template. Test the non-streamed body and streamed events against the same semantic result. The suite should assert token counts, finish reason, stop-token exclusion, tool-call arguments, schema validity, and error shape—not exact wall-clock chunk grouping.

For a slow consumer, cap the per-connection output buffer. When it fills, pause or cancel that stream without blocking the shared engine output path. After a disconnect, the request transitions to cancelling, future steps stop, in-flight output is ignored, and KV references eventually return to the baseline count.

For duplicate request ID r-17, choose and document one rule. Atlas rejects a second live attempt with conflict status; after a completed idempotent request, it may return the recorded terminal response for a retention window. Tool execution uses a separate idempotency key, because repeating generation and repeating an external action are not equivalent.

Run the suite against old and new engine revisions. Any difference is classified as intended API change, allowed numerical variation, or regression. “Both returned HTTP 200” is not conformance.

23. Benchmark card

A minimally credible card contains:

claim: candidate improves TTFT-qualified goodput for Atlas document traffic
model: exact artifact, tokenizer, precision, context configuration
system: engine commit, container digest, driver/runtime, kernel backends
hardware: device count and memory, CPU/NUMA, links, NIC, power policy
workload: published trace hash; open loop; input/output and prefix distributions
SLO: success, TTFT <= 600 ms, every ITL <= 150 ms, valid output
method: warm-up, cache state, run duration, repetitions, error/cancel policy
outputs: raw request events, server metrics, configuration, analysis revision
quality: task score and structured-output equivalence gate

Run baseline and candidate in randomized order at several offered loads. Keep failed and timed-out requests in the accounting. Report the goodput curve and confidence intervals, not only the best point.

If the second-day result moves, compare temperature, clock policy, background traffic, cache warmth, artifact hashes, and workload ordering. The correct response is to explain or bound the variance. Averaging two different regimes produces a precise number for no reproducible system.

24. Operations runbook

Symptom: p95 TTFT rises from 480 ms to 1.4 s while GPU utilization falls from 72 to 38 percent.

1. Check ingress and tokenizer queue age. If high, route around or scale that tier; rollback is removal of temporary capacity after the queue drains.
2. If those queues are normal, inspect engine admission age and reasons. A surge in remote-cache waits suggests a cache dependency, not insufficient GPU compute.
3. Compare scheduled prefill tokens with graph fallback and compilation events. If a new shape is compiling, stop canary traffic or disable that feature for the affected route, retaining the old artifact.
4. Check worker readiness and collective progress. Remove a failed group from routing before restarting it; draining preserves live state when possible.

At every branch, record the expected confirming signal. “Restart workers” is not a diagnosis and can destroy the evidence or cached state that explains the incident.

The failure drill delays KV transfers by 500 ms. Correct behavior is bounded transfer waiting, conditional recomputation or early rejection, cancellation cleanup, and recovery without leaked destination blocks. The dashboard must show stage queue age, transfer count and bytes, timeout reason, recomputation, and end-to-end goodput.

25. Architecture decision

Decision: serve Atlas on self-managed accelerator nodes using four-way tensor- parallel replicas, token-budget continuous batching, local prefix caching, and hybrid queue-plus-locality routing. Keep prefill and decode colocated initially; enable conditional disaggregation only for prompts above a measured transfer break-even. Use a managed API as an authenticated overflow route for supported requests, not as an invisible retry.

The choice follows the workload: interactive traffic needs bounded TTFT and ITL, document traffic benefits from prefix reuse, and the dense model does not fit on one device at baseline precision. Rejected alternatives are TP8, because the wider per-layer collective hurts low-concurrency decode; and unconditional disaggregation, because short prompts do not repay state transfer.

Security boundaries separate public generation, administrative weight/cache controls, and model artifacts. Tenant cache namespaces default to isolated. Prompts and KV state follow the same regional deletion policy. Releases pin the complete model and runtime identity and retain a rollback namespace.

Sensitivity changes the decision. If context length doubles, KV capacity or lower-precision KV may become binding. If prefix reuse falls below the measured threshold, locality-aware routing loses value. If peak traffic doubles for less than worker startup time, a warm pool or overflow capacity is necessary. If the TTFT objective tightens, separate prefill capacity may become worthwhile. An ADR is complete only when it names these review triggers and the evidence that would reopen the decision.

26. Security boundaries

Atlas uses six trust zones: public clients, the authenticated gateway, tenant- aware schedulers and workers, state stores, artifact supply, and privileged administration. Tools form a seventh zone because they receive model-proposed actions but hold authority outside the inference service.

The deletion proof starts with the request ID and its tenant namespace. Revoke new reads first, then invalidate local and distributed KV keys, encoder-feature keys, and any adapter- or parser-derived state. Delete prompt-bearing logs and traces or retain only fields covered by the declared telemetry policy. Remove the request from replay and benchmark datasets. Record acknowledgements from every cache replica and storage region rather than treating a control-plane request as proof of deletion.

Backups determine the upper bound. If encrypted backups expire after 30 days and cannot delete one object in place, the proof says “inaccessible now and physically expired within 30 days,” names the key-erasure mechanism, and tests that a restore cannot reintroduce live cache indexes or service-visible data. The exercise is complete only when every derivative has either a deletion receipt or a bounded expiry with an owner and verification procedure.

Appendix I. Debugging exercises

The debugging chapter’s walkthroughs are self-contained investigations. The practice exercise is to apply the five-step method to a new symptom: p99 E2E latency is 3× the p50, but p99 TTFT and p99 ITL are both within SLO.

The resolution: the symptom indicates variability in output length, not engine latency. Requests that generate 3× the median output length take 3× as long end-to-end while meeting per-token targets. This is a workload property, not an engine defect. The fix is to report E2E latency by output- length bucket and set per-bucket E2E expectations, or to impose output- length limits per request class.

Appendix H. Optimization Migration Guide

You have vLLM or SGLang running with default settings and real traffic. This guide walks each optimization from the book in the order you should evaluate it — starting from the changes with the highest impact and lowest risk, then moving toward changes that require more measurement and carry more interaction effects.

Each entry names the optimization, the chapter that explains it, the expected impact, the risk, and a concrete evaluation step. Do not apply them all at once. Apply one, measure with Chapter 23’s method, confirm the result, and then evaluate the next.

Phase 1: Free or near-free wins

These changes improve performance without meaningful risk and usually require only a configuration flag or a version upgrade.

1.1 Enable chunked prefill (Chapter 6)

Default state: Many deployments run with chunked prefill disabled or with a large chunk budget that effectively disables it.

What to do: Set --enable-chunked-prefill (vLLM) or verify chunked_prefill_size (SGLang). Start with a chunk budget of 512 tokens.

Expected impact: Reduces p99 ITL spikes caused by long prefills blocking decode steps. Typical improvement is 2–5x reduction in tail ITL.

Risk: Minimal. Prefill takes more steps to complete, increasing TTFT slightly for very long prompts. Measure both TTFT and ITL.

Evaluation: Run your workload with and without chunked prefill at your current load. Compare p50 and p99 for both TTFT and ITL. Appendix I’s first walkthrough demonstrates this measurement.

1.2 Enable prefix caching (Chapter 7)

Default state: Prefix caching is available but may be off by default depending on engine version.

What to do: Enable --enable-prefix-caching (vLLM) or the equivalent in SGLang. No other configuration needed.

Expected impact: If your traffic shares common system prompts or document prefixes, TTFT drops by the reusable fraction. A 2,000-token shared system prompt saves roughly 70 ms of prefill per cache hit.

Risk: Minimal. Cache uses the same KV blocks that would otherwise be allocated to new prefills. The allocator already handles this.

Evaluation: Monitor cache hit rate (vllm:prefix_cache_hit_rate or equivalent). If hit rate is below 5%, the optimization is real but small for your traffic pattern.

1.3 Verify attention backend selection (Chapter 8)

Default state: The engine selects an attention backend automatically. The default may not be optimal for your model and device.

What to do: Check which backend is selected in the startup logs. For most NVIDIA deployments, FlashAttention-2 or FlashInfer should be active. Verify with --attention-backend flag.

Expected impact: The right backend reduces kernel time. The wrong backend can add 10–30% overhead to attention-bound workloads.

Risk: None if measuring. A backend that fails will error at startup, not silently produce wrong results.

Evaluation: Run a fixed workload with each supported backend and compare step times. The winner depends on your model’s attention pattern, batch size, and sequence lengths.

Phase 2: Memory and throughput tuning

These changes require understanding your workload’s memory profile and may interact with each other.

2.1 Right-size max sequences and KV budget (Chapters 6–7)

Default state: max-num-seqs and related settings are often left at defaults that may be too high (causing preemption) or too low (leaving capacity unused).

What to do: Calculate your KV budget using the formula from Chapter 7:

available KV memory = GPU memory − weight shard − activation reserve − graph pool
KV tokens = available KV memory / (KV bytes per token / TP degree)
max sequences = KV tokens / average context length

Set max-num-seqs to a value where steady-state KV occupancy stays below 85% of available blocks.

Expected impact: Eliminates preemption storms — the most common cause of unexplained latency spikes. Appendix I’s first walkthrough shows a preemption storm reducing effective throughput by 40%.

Risk: Setting too low wastes capacity. Setting too high causes preemption. Measure KV block utilization under load.

Evaluation: Monitor vllm:gpu_cache_usage_perc under peak load. If it regularly exceeds 95%, reduce max-num-seqs. If it stays below 50%, you have headroom to increase it.

2.2 Enable CUDA graph capture (Chapter 9)

Default state: Usually enabled by default, but warmup may not cover all batch shapes your workload encounters.

What to do: Verify graphs are captured by checking startup logs for captured batch sizes. If you see repeated compilation warnings during serving, add those batch shapes to the warmup set.

Expected impact: Graph capture eliminates launch overhead. Decode steps become 10–20% faster. Uncaptured shapes fall back to eager mode, which shows as occasional latency spikes.

Risk: Each graph consumes GPU memory (typically 100–300 MB per bucket shape). Too many buckets can eat into KV headroom. Appendix I’s second walkthrough shows graph pool growth causing OOM.

Evaluation: Monitor reserved versus allocated CUDA memory. A growing gap between the two indicates graph pool growth for unseen shapes.

2.3 Evaluate quantization (Chapter 10)

Default state: Most deployments use BF16 or FP16 weights.

What to do: Test FP8 (E4M3) first if your hardware supports it (H100, MI300X). It halves weight memory with minimal quality loss for most tasks. If FP8 is unavailable, evaluate GPTQ-INT4 or AWQ-INT4.

Expected impact: FP8 roughly doubles KV headroom by halving weight memory. INT4 quadruples it. Decode throughput improves because weight reads are the bottleneck.

Risk: Quality degradation. Always measure task-specific quality (not just perplexity) before and after quantization on your evaluation set. The quantization chapter’s rule: measure quality and throughput together, never separately.

Evaluation: Run your quality evaluation suite at BF16 and at the target precision. If quality passes your gate, benchmark throughput and latency at your operating load.

Phase 3: Architecture changes

These changes affect the deployment topology and require more planning.

3.1 Tensor parallelism sizing (Chapter 13)

Default state: Many deployments default to TP matching GPU count without evaluating whether fewer ranks would suffice.

What to do: Use Chapter 13’s quick reference table. If your quantized model fits on fewer GPUs, test a narrower TP degree with the freed GPUs running as replicas instead.

Expected impact: Narrower TP means fewer collectives per step. TP2 instead of TP4 halves synchronization overhead. The freed GPUs as replicas add independent capacity.

Risk: Model must fit in the narrower TP group’s memory including KV headroom. Measure at peak batch size, not empty.

Evaluation: Compare per-request latency and fleet throughput at TP_N versus TP_{N/2} with 2× replicas. The winner depends on your batch sizes — Chapter 13’s worked example shows the analysis.

3.2 Speculative decoding (Chapter 11)

Default state: Disabled. Requires a draft model or multi-token prediction heads.

What to do: If your target model has MTP heads (e.g., DeepSeek models), enable them. Otherwise, find or train a small draft model (1–2B parameters) for your workload.

Expected impact: 1.5–2.5x decode speedup when acceptance rate exceeds 70%. Below 50% acceptance, speculative decoding hurts.

Risk: Draft model consumes additional memory. Acceptance rate is workload-dependent — creative generation accepts fewer tokens than formulaic tasks.

Evaluation: Enable speculation on a staging deployment and measure acceptance rate, TPOT, and total throughput. Appendix D’s speculative decoding checklist gives the thresholds.

3.3 Prefill/decode disaggregation (Chapter 15)

Default state: Colocated prefill and decode on the same workers.

What to do: Only evaluate this if you see ITL spikes correlated with prefill arrivals, or if your workload has highly skewed input/output ratios. Disaggregation requires a KV transfer mechanism and separate pool management.

Expected impact: Eliminates phase interference. Decode latency becomes independent of prefill load. Meaningful only when long-prompt interference is measurable.

Risk: Adds a transfer boundary, a second pool to scale, and coupled queue dynamics. The transfer itself costs time — Chapter 15 prices it at ~35 ms for a typical sequence. Short prompts may not repay this.

Evaluation: Measure ITL percentiles during mixed prefill/decode load. If ITL variance drops significantly with chunked prefill alone (Phase 1), disaggregation may not be needed.

Phase 4: Multi-tenant and scale-out

4.1 Adapter-aware routing (Chapter 12)

Applies if: You serve multiple LoRA adapters.

What to do: Configure adapter-aware routing that scores both load and adapter locality. Chapter 12’s worked example shows the routing score formula.

Expected impact: Reduces cold adapter loads by 4–5x compared to round-robin. Each cold load costs host-to-device transfer time.

4.2 Distributed caching (Chapter 16)

Applies if: You have multiple replicas and significant prefix overlap across them.

What to do: Evaluate cache-aware routing first (Chapter 17) before adding a distributed cache layer. Routing is simpler and often captures most of the value.

Expected impact: Depends entirely on your prefix reuse pattern. Measure cross-replica overlap before building infrastructure.

4.3 Autoscaling tuning (Chapter 24)

What to do: Scale on queue depth or waiting-request count, not GPU utilization. GPU utilization is a trailing indicator that can read high while requests queue.

Expected impact: Faster scale-up response to demand spikes. Appendix Appendix D’s autoscaling checklist gives the configuration procedure.

Evaluation order summary

Appendix I. Production Debugging Playbook

Chapter 23 teaches you to design experiments that produce trustworthy numbers, and Chapter 24 builds the observability that makes incidents diagnosable. This playbook applies both: it walks three real debugging sessions end-to-end, each starting from a symptom an operator would see in production and ending at a verified fix. The goal is not to catalogue every possible failure but to demonstrate the method – split the symptom into candidate causes, use evidence to eliminate branches, and verify the fix against the same measurement that raised the alarm.

Every walkthrough uses the Atlas constants: 140 GB of BF16 weights across a TP4 deployment (35 GB per rank), 320 KiB of KV state per token, 0.035 ms per prefill token, 45 ms decode step, 600 ms TTFT target, 150 ms ITL ceiling. Commands and metrics reference vLLM and SGLang at their pinned SHAs; the investigation method transfers to any engine.

Walkthrough 1: high TTFT under load

The symptom

Monday morning traffic ramp. Atlas’s p95 TTFT rises from its baseline of 480 ms to 1,400 ms – more than double the 600 ms target. The on-call engineer opens the GPU dashboard and sees utilization has dropped from 72 to 41 percent. The instinct is to add replicas. Resist it: falling utilization means accelerators are starving for work, not drowning in it. Adding capacity treats a symptom and hides the cause.

Step 1: router, queue, or engine?

The first split separates three regions of the request path. Each has a distinct signal.

flowchart LR
    subgraph Region
        direction LR
        R["Router / load balancer"] --> Q["Engine admission queue"]
        Q --> P["Prefill execution"]
    end
    R --> R1["Ingress latency histogram"]
    Q --> Q1["Queue age and depth"]
    P --> P1["Prefill duration histogram"]

Check the router first. If the load balancer is slow to assign requests, TTFT rises without any engine involvement. Pull the ingress-to-engine latency from the trace span:

# Query the last 15 minutes of router-to-engine latency from Prometheus
curl -s 'http://localhost:9090/api/v1/query?query=histogram_quantile(0.95,rate(router_forward_duration_seconds_bucket[5m]))'

If this value is under 20 ms, the router is not the bottleneck. Move on.

Step 2: locate the queue with vLLM metrics

vLLM exposes Prometheus metrics at its /metrics endpoint. Two are immediately diagnostic:

# How many requests are sitting in the waiting queue right now?
curl -s http://localhost:8000/metrics | grep 'vllm:num_requests_waiting'

# What does the TTFT distribution look like?
curl -s http://localhost:8000/metrics | grep 'vllm:time_to_first_token_seconds'

In this incident, vllm:num_requests_waiting reads 47 – far above the normal operating point of 3 to 8. The oldest waiting request has been queued for 1.1 seconds. That queue age directly consumes TTFT budget: a request that waits 1.1 seconds before prefill begins cannot meet a 600 ms TTFT target regardless of how fast prefill runs. The queue is the proximate cause. The question becomes: why is the queue backing up?

Two candidate mechanisms explain queue growth: prefill is taking longer than expected (requests drain slowly), or KV cache pressure is preventing admission (requests cannot enter the batch even though the GPU has time).

Step 3: profile the prefill path

Check whether prefill itself has slowed. Pull the engine’s prefill duration:

curl -s http://localhost:8000/metrics | grep 'vllm:e2e_request_latency_seconds'

Compare against the expected cost. A 2,000-token prompt on Atlas costs roughly 2000 * 0.035 = 70 ms of prefill compute. If chunked prefill is enabled with a 512-token chunk budget, that prompt crosses four engine steps of approximately 512 * 0.035 = 18 ms of prefill work each, interleaved with decode. The total wall-clock prefill time will be longer than 70 ms because each step also carries decode work, but the individual chunk durations should stay near 18 ms.

Three conditions inflate prefill time:

1. Long prompts without chunking. A 16,000-token prompt processed in one shot takes 16000 * 0.035 = 560 ms and holds the GPU for that entire duration, stalling every in-flight decode request. Check whether chunked prefill is enabled and whether the chunk budget is sized appropriately.

2. Missing CUDA graph capture. If the batch shape has not been captured as a graph, the engine falls back to eager execution. Check for graph-fallback log entries or the vllm:num_graph_captures counter not incrementing.

3. Interference from mixed steps. Under chunked prefill, decode tokens share each engine step with a prefill chunk. If the chunk budget is too large relative to the decode population, step time stretches and ITL suffers – but that would appear as an ITL problem, not a TTFT problem. In this case, the queue growth points elsewhere.

In this incident, prefill durations look normal. The queue is growing not because requests drain slowly, but because they are not being admitted.

Step 4: check KV cache pressure

This is the critical split. Pull the memory metrics:

curl -s http://localhost:8000/metrics | grep 'vllm:gpu_cache_usage_perc'
curl -s http://localhost:8000/metrics | grep 'vllm:num_preemptions_total'

KV cache usage reads 94 percent. Preemption count is climbing: 23 preemptions in the last five minutes, versus a baseline of zero. The mechanism is now clear.

Atlas’s KV budget per rank is approximately 35 GB (the accelerator memory minus the 35 GB weight shard, minus activation and graph overhead). At 320 KiB per token, that supports roughly 35 * 1024 * 1024 / 320 = 114,688 tokens of live KV state per rank. If the current batch has 40 active sequences averaging 2,800 tokens each, that is 40 * 2800 = 112,000 tokens – 97 percent of capacity.

When a new request arrives and no blocks are free, the scheduler must preempt: it evicts the KV state of one or more lower-priority requests, freeing their blocks, and admits the new request. The evicted requests re-enter the queue and must re-prefill from scratch when they are later re-admitted. This re-prefill inflates their TTFT enormously – a request that was 80 percent complete in decode loses all its KV state and starts over. Worse, the re-prefill consumes GPU time, slowing other admissions, creating a cascade.

The preemption storm explains both symptoms: TTFT rises because requests bounce between the queue and partial execution, and GPU utilization falls because the engine spends cycles re-computing KV state it already produced.

Step 5: the fix and verification

The immediate remediation is to tighten admission. Reduce the maximum concurrent sequences from 40 to 28, keeping KV occupancy below 80 percent of capacity:

# Restart with a lower max-num-seqs to leave KV headroom
python -m vllm.entrypoints.openai.api_server \
    --model atlas-70b \
    --tensor-parallel-size 4 \
    --max-num-seqs 28 \
    --enable-chunked-prefill

After the change, verify:

1. vllm:gpu_cache_usage_perc stays below 0.80.
2. vllm:num_preemptions_total stops climbing.
3. vllm:num_requests_waiting returns to the 3-to-8 range.
4. p95 TTFT drops below 600 ms.

The longer-term fix addresses why sequences grew long enough to exhaust the cache: check whether the output length distribution shifted (users submitting longer conversations), whether prefix caching is releasing blocks correctly, or whether the max-context-length setting is broader than the workload requires.

Walkthrough 2: memory pressure and OOM

The symptom

Over a 48-hour period, two of Atlas’s four TP4 workers restart with CUDA out-of-memory errors. The crashes happen at different times of day with no obvious correlation to request volume or type. Between crashes, GPU memory metrics look stable. The instinct is to blame a memory leak in user requests. That is almost never the cause.

Step 1: establish the memory budget

Before hunting leaks, know what the budget should be. For one Atlas rank:

The margin between the budget and the device limit is 2 to 5 GB. Any component that grows beyond its expected allocation will eventually trigger OOM. The question is: which component is growing?

Step 2: track allocated versus reserved

Start with the coarse signal:

# Snapshot GPU memory state across all ranks
nvidia-smi --query-gpu=index,memory.used,memory.free,memory.total \
    --format=csv,noheader,nounits

This shows total memory consumption but does not distinguish PyTorch allocations from CUDA driver state. For finer granularity, enable PyTorch’s memory snapshot:

import torch

# Enable memory history tracking (do this before model load)
torch.cuda.memory._record_memory_history(max_entries=100000)

# ... run workload ...

# Dump snapshot to file for analysis
torch.cuda.memory._dump_snapshot("memory_snapshot.pickle")

Load the snapshot in PyTorch’s memory visualizer (torch.cuda.memory._snapshot()) to see allocation timelines. Two patterns distinguish the common causes:

• A leak appears as a monotonic increase in allocated memory that never returns to baseline, even when the request queue is empty.
• A pool growth appears as step increases in reserved (but not necessarily allocated) memory that coincide with specific events.

In this incident, the snapshot shows reserved memory growing in discrete 2 GB jumps roughly every 8 to 12 hours.

Step 3: check for KV block leaks

A KV block leak occurs when a request completes but its blocks are not returned to the free pool. The block manager’s reference count stays nonzero, and the blocks remain allocated forever. Over hours, the free pool shrinks.

Check the free block count over time:

# Track free blocks via the metrics endpoint
curl -s http://localhost:8000/metrics | grep 'vllm:gpu_cache_usage_perc'

If cache usage ratchets upward even during low-traffic periods (when sequences complete faster than they arrive), blocks are leaking. In this incident, cache usage returns to baseline during off-peak hours. The leak is not in the KV pool.

Step 4: check CUDA graph pool growth

CUDA graphs capture a fixed sequence of GPU operations for replay without CPU launch overhead. Each captured graph allocates a private memory pool for the tensors it uses during execution. The pool is sized to the captured shape – batch size, sequence length, and intermediate buffer requirements.

The critical detail: if a batch shape arrives at runtime that was not captured during warmup, the engine must either fall back to eager execution or capture a new graph on the fly. On-the-fly capture allocates a new graph pool. If the shape is rare, the pool sits mostly idle, consuming memory without proportionate benefit.

Check for graph captures after startup:

# Look for graph capture events in the engine log
grep -i "capturing\|cuda graph\|graph capture" /var/log/vllm/engine.log

In this incident, the logs show new graph captures at irregular intervals – each coinciding with one of the discrete memory jumps in the snapshot. The trigger is an unusual batch composition: when the scheduler happens to assemble a batch of 17 decode tokens plus a 384-token prefill chunk (a shape not seen during warmup’s graph capture sweep), the engine captures a new graph. Each capture allocates roughly 1.5 to 2.5 GB of graph pool memory that persists for the process lifetime.

Over 48 hours, three or four such captures accumulate 6 to 10 GB of graph pool memory beyond the startup budget, consuming the 2 to 5 GB margin and eventually triggering OOM on the next allocation spike.

Step 5: the fix and verification

The fix has two parts.

First, expand the warmup capture set to cover all reachable batch shapes. The graph capture sweep should enumerate the combinations of decode batch sizes and prefill chunk sizes that the scheduler can actually produce, not only the “standard” batch sizes:

# In the engine configuration, specify explicit capture shapes
# that cover the scheduler's actual output range
--enforce-eager  # Temporary: disable graphs to stop the bleeding

Then, after computing the full set of reachable shapes from the scheduler’s budget and chunk configuration, re-enable graph capture with an explicit shape list or rely on the engine’s padded capture buckets (Chapter 9). If the engine supports a maximum graph pool size, set it:

# Limit graph memory to prevent unbounded growth
# (engine-specific; check documentation for the exact flag)
export VLLM_GRAPH_RESERVED_MEM=4GiB

Second, add a memory-growth alert. The signal is not total memory usage (which fluctuates with batch load) but the gap between PyTorch’s reserved_memory and allocated_memory. A growing gap that does not shrink during idle periods indicates pool fragmentation or graph accumulation:

# Monitor the reserved-allocated gap
python -c "
import torch
for i in range(torch.cuda.device_count()):
    r = torch.cuda.memory_reserved(i) / 1e9
    a = torch.cuda.memory_allocated(i) / 1e9
    print(f'GPU {i}: reserved={r:.1f}GB allocated={a:.1f}GB gap={r-a:.1f}GB')
"

Verification: after restarting with the expanded capture set, confirm that no new graph captures appear in the log after warmup completes, and that the reserved-allocated gap remains stable over 72 hours.

Walkthrough 3: tail ITL spikes

The symptom

Atlas’s median inter-token latency holds steady at 48 ms – healthy, given the 45 ms decode step plus sampling and streaming overhead. But p99 ITL spikes to 300 ms or higher, well above the 150 ms ceiling. The spikes are intermittent and do not correlate with request volume in an obvious way. The instinct is to profile the decode kernel. That is almost certainly not the problem – a kernel that is fast at the median does not become 6x slower at the tail without a discontinuity.

Step 1: correlate spikes with batch composition

ITL measures the gap between consecutive tokens in a single response stream. A spike means one particular engine step took much longer than usual. The question is whether the spike belongs to the model (the compute was slow) or to the batch (the step included extra work).

Under chunked prefill, each engine step may contain a mix of decode tokens and a prefill chunk. The prefill chunk adds compute: a 512-token chunk costs roughly 512 * 0.035 = 18 ms of additional work in the step. If the chunk budget is set to 1,024 tokens, the chunk alone adds 1024 * 0.035 = 36 ms, stretching the step from 45 ms to approximately 81 ms. Add scheduling overhead, attention over the growing context, and the decode portion: a step near 90 ms is plausible, but 300 ms is not explained by a single standard chunk.

Pull the per-step composition from the engine’s iteration metrics:

# Check the token composition of recent engine steps
curl -s http://localhost:8000/metrics | grep 'vllm:num_tokens_prefill'
curl -s http://localhost:8000/metrics | grep 'vllm:num_tokens_decode'

In this incident, the spikes correlate with steps where the scheduler admits a very long prefill chunk – 4,096 tokens or more from a new request with a long prompt, processed without chunking because the request’s priority forced immediate admission. That chunk costs 4096 * 0.035 = 143 ms of prefill compute, pushing the mixed step past 200 ms. If two such admissions coincide, the step reaches 300+ ms.

Every decode-phase request in that step sees the entire step duration as its ITL for that token. The spike is not in their computation; it is in the time they waited for the step to complete.

Step 2: profile individual engine steps

To confirm, capture a short trace with PyTorch profiler during a period when spikes are occurring:

from torch.profiler import profile, ProfilerActivity, schedule

with profile(
    activities=[ProfilerActivity.CPU, ProfilerActivity.CUDA],
    schedule=schedule(wait=1, warmup=1, active=5, repeat=1),
    on_trace_ready=torch.profiler.tensorboard_trace_handler('./traces'),
    record_shapes=True,
    with_stack=True,
) as prof:
    for step in range(20):
        engine.step()
        prof.step()

Load the trace in TensorBoard or Chrome’s chrome://tracing. Look for the step that corresponds to the ITL spike. In the CUDA stream, you will see the attention kernels for the prefill chunk dominating the step – their combined duration matches the expected chunk_tokens * 0.035 ms cost, confirming that the compute is correct but the chunk is too large.

Step 3: check for GC pauses and CPU-side bottlenecks

Not all ITL spikes come from the GPU. Python’s garbage collector can pause the engine’s main loop, and CPU-side output processing (detokenization, streaming, sampling) can hold up the next step.

Check for GC pauses:

import gc

# Enable GC debugging to see collection events
gc.set_debug(gc.DEBUG_STATS)

If GC collections correlate with ITL spikes (visible in logs as “gc: collecting generation 2” events taking 10+ ms), the fix is to disable automatic GC and collect manually between steps or during idle periods:

gc.disable()
# Collect explicitly during known idle points

In this incident, GC pauses account for a few spikes in the 160 to 180 ms range but not the 300+ ms outliers. The primary cause remains the oversized prefill chunks.

Step 4: check for collective stragglers in TP groups

In a TP4 deployment, every engine step ends with an all-reduce across four ranks. The step completes only when the slowest rank finishes. A rank that is slow – due to thermal throttling, PCIe contention, memory bandwidth saturation, or an unrelated process on the host – extends the step for all ranks.

Check for rank imbalance:

# Compare per-GPU utilization and clock speeds
nvidia-smi --query-gpu=index,clocks.current.sm,utilization.gpu,temperature.gpu \
    --format=csv,noheader