GQA KV Cache Memory Mechanics
Parent: 16 GB VRAM residency budgeting · Published reference · snapshot 2026-09-30
↓ Facts as markdownall context files
Source-anchored research for GQA KV Cache Memory Mechanics. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Structure and components
- For Qwen2-14B with 48 layers, 8 KV heads, and 128 head dimension at FP16, KV cache consumes exactly 196.6 KB per token, or 3.14 GB for a 16,384-token context window. (high confidence) [source]
Measurements and reference values
- FP8 KV cache quantization reduces per-token KV memory by 50% down to 98.3 KB per token, permitting up to 32,768 context tokens in 3.14 GB on 16 GB accelerators. (medium confidence) [source]
Facts and statements
- In GQA architectures such as Qwen2-14B and Llama-3.1-8B, KV cache memory footprint scales as 2 * num_layers * num_kv_heads * head_dim * precision_bytes per context token. (high confidence) [source]
Children
- No children recorded.