The sheet has the inputs (read from Kimi K2.6's Hugging Face model card and config.json), the derived numbers, a table of machines, and the formulas so you can redo it for any other model. Here is what is in it.
The inputs
| Input | Value | Where it comes from |
|---|---|---|
| Total parameters | 1 trillion | model card |
| Active per token | 32 billion | model card |
| Experts | 384, with 8 used per token | config.json |
| Layers | 61 | config.json |
| KV compression (MLA) | kv_lora_rank 512 + rope 64 | config.json |
| Checkpoint size | 595.2 GB (4-bit weights) | Hugging Face repo size |
| Context | 256K tokens (262,144) | model card |
Step 1: the weights
- At 16 bits (2 bytes per parameter): 1 trillion × 2 = 2 TB. An H100 has 80 GB, so that is 25 H100s just to hold the weights.
- Kimi is trained to run at 4 bits, half a byte per weight. The real checkpoint is 595.2 GB.
Mixture of experts does not make the model smaller. All 384 experts must sit in memory. It makes it faster: each token only runs through 8 experts, about 32 billion parameters instead of a trillion.
Step 2: the KV cache
Every token in a conversation leaves keys and values behind for the attention layers. Kimi uses multi-head latent attention (MLA), which squeezes that into one small vector per layer:
- 61 layers × (512 + 64) values × 2 bytes = 70,272 bytes per token, about 70 KB.
- One full 256K-token chat: 70,272 × 262,144 = 18.4 GB.
Step 3: fit it on a machine
| Machine | GPU memory | Left after weights | Full 256K chats |
|---|---|---|---|
| 1× H100 | 80 GB | none | 0 |
| 4× H200 | 564 GB | none (31 GB short) | 0 |
| 8× H100 | 640 GB | 44.8 GB | 2, barely |
| 8× H200 | 1,128 GB | 532.8 GB | 28 |
| 8× B200 (DGX) | 1,440 GB | 844.8 GB | 45 |
That is the reel's point: once the weights fit, the extra memory is what lets you serve more people. Buy memory, not compute.
Size any model yourself
Weights (GB) = parameters × bits ÷ 8 ÷ 1e9
KV per token = 2 × layers × kv_heads × head_dim × bytes (standard / GQA attention)
KV per token = layers × (kv_lora_rank + rope_dim) × bytes (MLA: DeepSeek, Kimi)
Chats that fit = (GPU memory − weights) ÷ (KV per token × context length)All of layers, kv_heads, head_dim and kv_lora_rank are in the model's config.json on Hugging Face.
What the sheet ignores
Weights and KV cache only. A real serving stack also needs memory for activations, CUDA graphs and the framework itself, usually a few GB per GPU, and most servers do not let the cache fill every last byte. Treat "2 chats on 8× H100" as "it fits, just", not as a plan.