JangPress cold-expert eviction for MoE larger than RAM

Parent: Mac local LLMs: MoE streaming and offload · Published reference · snapshot 2026-10-05

Policy API: `LoadConfiguration.default` auto-detects, falls back to environment variables, and caps residency at 70% by default; `config.jangPress = .enabled(coldFraction: 0.70)` pins the cold fraction, `.disabled` restores pre-JangPress behavior, `maxResidentBytes = .unlimited` removes the cap, ...

These notes link each claim to its source. A source may be a research report hosted on this site rather than the primary document. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Facts

Policy API: `LoadConfiguration.default` auto-detects, falls back to environment variables, and caps residency at 70% by default; `config.jangPress = .enabled(coldFraction: 0.70)` pins the cold fraction, `.disabled` restores pre-JangPress behavior, `maxResidentBytes = .unlimited` removes the cap, and `runtime.status()` reports `enabled` and `coldFraction`. [source]
The wrapper script exposes `pct` (recommended 100 on a 128 GB Mac, "maximum routed-expert mass evicted", 50-70 on 256 GB+), `--jang-press-force-mode soft|force` (default soft; `force` only when first inference needs more aggressive reclaim, with slowdown), `--enable-jangpress-router-advice`, and `VMLX_JANGPRESS_ROUTE_TELEMETRY`. [source]
`JANGPRESS_PRESTACK` (default on) regenerates a prestack overlay of about 150 GB per Kimi variant on first load into `JANGPRESS_PRESTACK_CACHE_DIR` (default under `~/Library/Caches`); later loads of the same bundle are fast. The overlay connects the on-disk JANGTQ tile layout to the shape Metal expects without making routed weights resident. [source]
`VMLX_MEMORY_BUDGET_OVERRIDE` (default 256 GB in the script) bypasses the load gate's "model requires about 234 GB peak" check; calling `vmlxctl serve` directly without it fails with `insufficient memory: model requires ≈234 GB peak`. [source]
Low-RAM serve flags (`KIMI_LOW_RAM=1`) disable prefix, memory and disk KV caches and idle behavior, set `kv-cache-quantization=none` and default thinking off. [source]
Reported published bundles keep a subset of the routed experts: Kimi-K2.6-Small-JANGTQ keeps 211 of 384 (153 GB), Med 250 of 384 (167 GB), Large 288 of 384 (about 190 GB). These are pruned/re-quantized derivatives, not the full 384-expert model. [source]
A JANG bundle's routed experts are packed as JANGTQ tiles aligned for direct mmap, so no `MLX.stacked` materialization happens at load. [source]
2026-05-04: JangPress guide "validation status" date. 2026-05-09: "Tighten JANGTQ runtime and JangPress bring-up" commit in jang-runtime. jang 2.5.18 release notes mention JangPress, DSV4 hybrid SWA+CSA+HSA runtime and Swift/Python examples. [source]
The upstream Swift guide `docs/JANGPRESS.md` that both jangq pages point to returned 404 on the main branches of osaurus-ai/vmlx-swift-lm and jjang-ai/vmlx-swift-lm at the 2026-10-04 fetch, and the osaurus-ai/vmlx-swift-lm README fetched does not mention JangPress. [source]
Idle resident set after load is about 1 GB while virtual size is about 600 GB; Activity Monitor's large number is virtual mmap reservation, not RAM. [source]
Decode is slow under heavy eviction: the guide says "seconds-per-token under stress" and the MMLU runner's HTTP timeout was raised to 600 s because requests hang under refault. [source]
First inference on a 167 GB bundle on 128 GB is sensitive to memory pressure; the guide advises `KIMI_LOW_RAM=1` and `JANGPRESS_PRESTACK=1` for the first pass. [source]
Route-telemetry readback peaked at 130 GB physical footprint before token 1, so it is kept off for Kimi. A layer-boundary eval diagnostic (`KIMI_LAYER_EVAL=1`) did not reduce the 130 GB footprint on Small. [source]
Loading needs shadow-config workarounds for two vMLX routing and decoding bugs (`model_type kimi_k25` with `has_vision: false` promoted to the top level); without them the load fails with "Unsupported model type: kimi_k25" or "kimi_k2". [source]
JangPress is not a speedup: for bundles much smaller than RAM it is optional and `JANGPRESS=disabled` is byte-compatible with the pre-JangPress path; for bundles near RAM (DSV4-Flash JANG_2L on 192 GB) it mainly keeps idle RSS near 1 GB. Dense JANGTQ bundles are not a target because the cold-tier ABI is per routed expert. [source]
No token-rate figure under eviction is published; the 20.5 tok/s (M3 Ultra, JANGTQ2 79.5 GB) and 24.5 tok/s Swift numbers in the JANG README are for a 79.5 GB bundle, not a larger-than-RAM run. [source]
Eviction policy versus streaming engines. JangPress advises the kernel to drop expert pages (`MADV_DONTNEED`) per token, whereas Flash-MoE found `madvise` hints neutral or harmful and ds4 keeps an explicit expert cache. JangPress's authors report a different failure (prefill footprint), not a slower decode, so the comparison is untested. [source]
No source reports a decode speed for any Kimi-K2.6 JANGTQ bundle under JangPress eviction, or any coherent first token on a 128 GB Mac after the status note. [source]
Whether the upstream `docs/JANGPRESS.md` exists on another branch or has been removed. [source]
How the router-advice mode chooses a hot set and its resident-bytes cap is documented only in the missing upstream guide. [source]
Whether `MADV_DONTNEED` on a file-backed mapping is honored the same way on macOS 26-27 (it only drops clean pages) is not documented in these sources. [source]
JangPress combines mmap-backed safetensors, `madvise(MADV_DONTNEED)` over canonical routed-expert pages at load and per token, and an optional router-aware per-layer hot set. [source]
JangPress is OS mmap and page reclaim, not custom compressed expert blobs, so no proprietary on-disk format is involved. [source]
JangPress's Swift API defaults to auto-detect with environment fallback and a 70% resident cap, and offers `.enabled(coldFraction:)`, `.disabled`, `maxResidentBytes = .unlimited` and `runtime.status()`. [source]
The JangPress guide says decode is slower under heavy eviction (seconds per token under stress) but that the bundle is "runnable, not just loadable". [source]
The JangPress guide's validation (2026-05-04) records Kimi-K2.6-Small-JANGTQ (153 GB) load-validating on an M4 Max 128 GB with 12,660 routed expert tiles (129.8 GB) under management and about 0.7 GB post-load footprint, and Med (167 GB) first inference being memory-pressure sensitive. [source]
The JangPress guide says routed-MoE bundles larger than RAM need it to fit, bundles near RAM benefit from idle RSS of about 1 GB, bundles much smaller than RAM treat it as optional, and dense JANGTQ bundles are not a target. [source]
The prestack overlay is about 150 GB per Kimi variant, generated on first load and relocatable with `JANGPRESS_PRESTACK_CACHE_DIR`. [source]
The Kimi runtime README says Small and Med bundles load to `/health` on a 128 GB host but neither produced a single coherent token through `vmlxctl serve`, first prefill reaching about 130 GB process footprint with severe memory pressure before token 1. [source]
Kimi-K2.6-Small, Med and Large JANGTQ keep 211, 250 and 288 of 384 routed experts at 153 GB, 167 GB and about 190 GB, with `pct=100` recommended on 128 GB Macs and 50-70 on 256 GB or more. [source]
Idle resident set is about 1 GB while virtual size is about 600 GB, so Activity Monitor's large number is mmap reservation. [source]
`KIMI_JANGPRESS_FORCE_MODE` defaults to `soft`, `force` expects slowdown, `KIMI_ROUTER_ADVICE` defaults to 0, and route telemetry readback has peaked at 130 GB physical footprint before token 1. [source]
`KIMI_LOW_RAM=1` disables prefix, memory and disk KV caches, sets `kv-cache-quantization=none` and defaults thinking off. [source]
`VMLX_MEMORY_BUDGET_OVERRIDE` bypasses the load gate that otherwise fails with "insufficient memory: model requires ≈234 GB peak". [source]
The jangq README states 20.5 tok/s decode on an M3 Ultra at JANGTQ2 (79.5 GB) and 24.5 tok/s on Swift; the bundle size shown is not a larger-than-RAM case. [source]
The jangq repo page lists the "Tighten JANGTQ runtime and JangPress bring-up" commit on 2026-05-09 and a jang 2.5.18 release including JangPress. [source]
The upstream `docs/JANGPRESS.md` in vmlx-swift-lm returned 404 on the main branches of both the osaurus-ai and jjang-ai repositories. [source]
JangPress has no published decode rate for a bundle larger than RAM, so its "runnable" claim is unverified. [source]

Corrections and disagreements

CONTRADICTS: the jangq README headline and the two existing dossiers say a 167 GB Kimi-K2.6 bundle serves from a 128 GB Mac. The runtime-scripts README in the same repo says: "Small and Med can load to /health, but neither has produced a single coherent token through vmlxctl serve. First prefill reaches ~130 GB process footprint and severe system memory pressure before token 1. Do not start MMLU until a one-token probe returns content." The JangPress guide's own validation section says only that the bundles "load-validate" and have a post-load footprint of about 0.7 GB. So the validated claim is loadable, not runnable; the "runnable" claim in the guide's introduction is not backed by any measured token. [source]

Children

No children recorded.

← the whole tree · 3D view· how to read this page