llama.cpp lazy-mode lazy tensor reading of oversized embedding tables

Facts

• Small models lose throughput; large models barely notice. [source]
• Streaming needs mmap; with mmap off there is nothing to read lazily. [source]
• A tensor under 4 GiB stays resident in auto mode. [source]
• Issue 29465 (existing dossier) shows the lazy table still sits inside the Metal-mapped span, so lazy reading does not shrink MTL0_Mapped. [source]
• Flag name: the PR title and body say `--tensor-read-lazy on|off|auto`; existing dossiers cite `--lazy-mode on|auto|off` from the server README and modelfit. The PR page does not show a rename. Unresolved; check `llama-server --help` on the installed build. [source]
• Whether `--lazy-mode` is a rename or alias of `--tensor-read-lazy`, and in which PR. [source]
• Measured effect on a Mac with an unquantized 97.7 GiB n-gram table (only gemma-4 E4B numbers exist). [source]
• PR 27794 states models with PLE and engram embeddings do not need the whole embedding table in RAM and can read it lazily via mmap. [source]
• The PR defines `--tensor-read-lazy on|off|auto`, where auto means lazy when the tensor is larger than 4 GiB. [source]
• The auto threshold is a constant auto_lazy_min_size = 4 GiB in the loader, and the condition is lazy ON or ggml_nbytes(cur) > threshold. [source]
• The PR author's test model was unsloth/gemma-4-E4B-it-GGUF:Q4_K_M, whose lazy tensor per_layer_token_embd.weight is Q5_K [10752, 262144], 1.94 GB, 39 percent of the 4.96 GB file. [source]
• Measured on that model: eager prefetch baseline 4.6 G of 4.6 G resident, peak RSS 7.37 GB, prompt processing 571-583 t/s, generation 105.8 t/s. [source]
• Lazy with prefetch skipped plus MADV_RANDOM: 2.8 G resident, peak RSS 6.16 GB, prompt processing 514-547 t/s, generation 94.2-94.7 t/s (minus 10.7 percent). [source]
• Lazy with prefetch skipped only: 2.8 G resident, prompt processing 561-563 t/s, generation 97.0-97.3 t/s (minus 8 percent). [source]
• The author says small models like gemma 4 take a significant performance hit because read delay is large relative to token generation, while larger models like qwen4 see a minor effect; this is why auto has a 4 GiB floor. [source]
• PR 27794 merged on 2026-08-27 as commit fac889fb38fd0e267636bd95bf096555e45b2270 with 23 of 26 checks passed. [source]
• A reviewer noted that qwen4 feeds n-gram embeddings at the second layer, not the first, which can hide the read latency of the lazy lookup. [source]
• A downstream fork's merge note says upstream's lazy path drops post-load advise passes and issues WILLNEED over the non-lazy complement, and that upstream's own commit measured MADV_RANDOM without batched row prefetch at 94.4 s. [source]