DeepSeek MLA: 70 GB of KV Cache at 1M Tokens

DeepSeek-V3’s published config.json caches 576 numbers per token, per layer. Across its 61 layers in bf16, that is 70,272 bytes — about 70 KB of KV cache for every token sitting in the window. Fill a million-token context with a single sequence and you are holding roughly 70 GB of cache, before model weights, before activations, before a second concurrent request.

That number, not the one on the spec sheet, is what “million-token context” costs you.

We wrote this because the DeepSeek-V4 conversation is running ahead of the artifacts. We could not find a published V4 model card, paper, or config file to check any claim against, so this article does not tell you what V4 does. It works the arithmetic from the configs that are public, and tells you which four numbers to read first when V4’s config lands.

Where 70 KB per token comes from

Multi-head Latent Attention (MLA), introduced in DeepSeek-V2 in May 2024, does not cache keys and values per head. It projects them down into a single low-rank latent vector and caches that instead. In V3’s config, kv_lora_rank is 512. Alongside it sits a 64-dim decoupled RoPE key (qk_rope_head_dim), which cannot be folded into the compression because rotary position has to be applied before the low-rank projection. 512 + 64 = 576 elements cached per token per layer.

The rest is multiplication: 576 x 2 bytes x 61 layers = 70,272 bytes per token.

Compare that to a model that already uses grouped-query attention. Llama 3.1 70B has 80 layers, 8 KV heads, and a head dim of 128. Keys and values together are 2 x 8 x 128 = 2,048 elements per layer per token, x 80 layers x 2 bytes = 327,680 bytes. That is about 320 KB per token, or 4.7x DeepSeek-V3’s figure — against a model that is not naive multi-head attention. DeepSeek’s V2 paper reported a 93.3% KV-cache reduction relative to its own 67B dense predecessor, which used full MHA.

Project both to a million tokens and the difference stops being an optimization detail:

• DeepSeek-V3 layout: ~70 GB of cache. That fits on one H200, tightly.
• Llama 3.1 70B layout: ~328 GB. That is a multi-GPU sharding problem for one user’s one request.

This is the actual reason MLA matters, and it is the thing to check on any model that advertises a very long window. A context length is a claim about positional encoding. Bytes per token is a claim about whether you can afford to use it.

Three things break before you reach the limit

Prefill compute, not memory. MLA cuts cache bytes. It does not make attention cheaper to compute over a long prompt, which still scales quadratically with sequence length. A million-token prefill is a time-to-first-token problem measured in tens of seconds even on good hardware. Prompt caching rescues this only when your prefix is stable across turns — in an agent loop that appends tool output every step, the cache invalidates constantly and you pay full prefill repeatedly.

Advertised context is not effective context. Needle-in-a-haystack is close to saturated and is now a weak signal. Benchmarks that require joining two facts, tracking a variable through many updates, or aggregating across the window — the RULER family and its successors — show degradation well before the advertised ceiling on every model tested. Treat a vendor’s single needle score as evidence of nothing except that lookup works.

Here is the failure mode that costs real debugging time, and it is worth understanding properly rather than skimming. You paste a large repository into a long window and ask about a function. The model finds it. It also finds a vendored or duplicated copy of the same symbol elsewhere in the window, and silently answers using the stale definition. Nothing errors. Retrieval succeeded; disambiguation failed. You get a confident, wrong answer about your own code, and the only tell is that the line numbers do not match. Small windows fail loudly by omitting context. Large windows fail quietly by including too much of it.

Price steps, not slopes. Gemini 2.5 Pro and Anthropic’s 1M-token Sonnet beta both charge a higher per-token rate above 200K tokens. Check the current pricing page rather than assuming the headline rate — a 1M-token prompt resent every turn is the expensive shape, not the one-off analysis.

What to use for codebase work right now

For almost everything a developer does day to day, retrieval against a 128K–200K window beats dumping the repository into a million-token prompt. It is faster, cheaper, and fails in the direction you can see.

The condition that flips it: when the answer depends on a global property that no individual chunk contains. Auditing every call site of a deprecated API, tracing a full call graph, or planning a migration that touches hundreds of files are questions retrieval genuinely cannot answer, because there is no chunk to retrieve — the answer is the set of all chunks. Those are worth the long window and the price step. Everything else is not.

If you want the long window today, Gemini 2.5 Pro is what we would reach for at that size on cost-per-token grounds, not a DeepSeek checkpoint, until a V4 config is public and testable.

What we did not test, and what to read when V4 ships

We did not run DeepSeek-V4. We found no weights, config, or paper to run. We also did not benchmark V3 at a million tokens — its published limit is 128K, so the 70 GB figure above is an extrapolation of its cache layout, not a measurement of a shipping product.

When a V4 config appears, four fields answer most of the question before anyone publishes a benchmark:

1. kv_lora_rank + qk_rope_head_dim — bytes cached per token per layer.
2. num_hidden_layers — the multiplier on that.
3. max_position_embeddings — the claimed window, which is the least informative of the four.
4. Whether the attention is sparse or selective. DeepSeek published its own native sparse attention work in February 2025, and some form of sparsity is the honest tell that a million-token window is an architectural capability rather than a positional-extension trick applied to a model trained on far shorter sequences.

If the first three multiply out to something that does not fit on the hardware you have, the fourth is the only thing that can save it.