TurboQuant

How Google Compresses LLM Memory 6× Without Losing a Single Answer

Based on TurboQuant: Online Vector Quantization with Near-Optimal Distortion Rate (Google Research)

TurboQuant overview

▦ 17 sections

⚙ 6 parts

Part I — Setting the Stage

01 The Memory Wall — Why You Should Care 02 Transformers — The 10,000 Foot View 03 Attention — The Heart of the Transformer

Part II — The KV Cache Problem

04 Why the KV Cache Exists 05 Why Compressing the KV Cache Is Hard

Part III — Quantization Fundamentals

06 Scalar Quantization 101 07 From Scalar to Vector Quantization — The Independence Question

Part IV — The Algorithm

08 The Random Rotation — Making Any Vector Predictable 09 The Distortion Guarantee — How Good Is This? 10 The Bias Problem — When Minimizing MSE Isn't Enough 11 The Two-Stage Fix — QJL on the Residual

Part V — Does It Actually Work?

12 Empirical Validation — Theory Meets Practice 13 KV Cache Experiments — The Headlines 14 Nearest Neighbor Search — The Bonus Application

Part VI — Practical & Closing

15 What the Paper Doesn't Tell You 16 Where This Fits in the Inference Stack 17 Closing — The Three-Step Summary

Begin Presentation →