MLX KV Cache Quantization and Zero-Copy Pointer Passing
Parent: Local Inference Acceleration: Memory Architectures, Metal Kernels, and Latency Optimization · Published reference · snapshot 2026-09-30
↓ Facts as markdownall context files
Source-anchored research for MLX KV Cache Quantization and Zero-Copy Pointer Passing. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Measurements and reference values
- Quantizing KV cache tensors to 4-bit or 8-bit in MLX reduces context memory growth from ~9.0 GB to ~2.5 GB at 64k tokens on 35B parameter models with negligible loss in retrieval recall. (high confidence) [source]
Facts and statements
- MLX allocates tensor buffers directly into unified memory pages mapped with shared CPU-GPU storage modes, eliminating DMA transfers and serialization overhead. (high confidence) [source]
Children
- No children recorded.