KV checkpoint eviction scoring by tokens per byte with hit decay

Parent: Mac local LLMs: Prompt cache and persistent KV · Published reference · snapshot 2026-10-05

Upstream `create_checkpoint()` runs three steps in order: (1) when the list is full or one short of full, erase checkpoints that sit within `checkpoint_min_step` tokens after an earlier kept one, unless the current task created them; (2) while the list is still full, erase the front (oldest) chec...

These notes link each claim to its source. A source may be a research report hosted on this site rather than the primary document. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Facts

Upstream `create_checkpoint()` runs three steps in order: (1) when the list is full or one short of full, erase checkpoints that sit within `checkpoint_min_step` tokens after an earlier kept one, unless the current task created them; (2) while the list is still full, erase the front (oldest) checkpoint with the log `erasing old context checkpoint`; (3) erase any checkpoint at the same `n_tokens` as the new one (log `superseding context checkpoint`). [source]
The checkpoint struct carries only `id_task`, `pos_min`, `pos_max`, `n_tokens` and state data; no hit counter, byte-efficiency field or decay timestamp appears in the creation path. [source]
The restore search walks checkpoints newest to oldest and takes the first whose `pos_max <= pos_next` and (`pos_min < pos_min_thold` or `pos_min == 0`). The first match wins, so no hit is recorded. [source]
Checkpoints beyond the resume point are erased right after the search (`erased invalidated context checkpoint`, condition `pos_max > pos_next`). [source]
Hybrid checkpoints are nearly constant size (149.626 MiB at 364 and 2,800 tokens), so any tokens-per-byte score would rank mostly by position for a hybrid model. [source]
Before PR 25472 the loop erased only the front. PR 25472 (2026-07-09) added min-step proximity eviction. [source]
PR 25592 (opened 2026-07-12 by krim404, still open) originally proposed "eviction drops the checkpoint closest to its neighbor instead of the oldest one so early anchors survive edited/compacted histories". After rebase on 2026-07-14 it states PRs 25472 and 25649 landed the min-step eviction and near-end checkpointing upstream, and it carries only exact-position restore, prefix-based restore and stale-checkpoint invalidation. [source]
PR 25420 (merged 2026-07-09 as 64c8b7d) made prompt-batch splitting at user messages respect min-step; it fixes issues 25213 and 25320 (prefill slowdowns). [source]
Issue 25023 (opened 2026-06-25): with `--ctx-checkpoints 8 --checkpoint-min-step 8192` and ~138K context, the `is_last_user_message` bypass created a checkpoint every turn at about 1K spacing, so eight checkpoints clustered in 74K to 83K; a later request at 30,646 tokens erased all of them and reprocessed 40K tokens in 18.2 s. PR 25472 closed it. [source]
Oldest-first removal drops the early anchor that survives a compaction or an edit near the start, which is the case the 25592 neighbor-distance idea targeted. [source]
Position-based (upstream) versus value-based (fork claim): oldest-first favors recency; a tokens-per-byte score favors long, cheap checkpoints. Both are valid for different workloads and neither has a measurement in the sources. [source]
Whether a fork implements tokens-per-byte with hit decay, and its decay constant. [source]
Whether the bypass in step (1), which skips checkpoints created by the current task, can leave two near-end checkpoints (n_ubatch+4 and 4 tokens before the end) both alive by design; the code comment says yes. [source]
Upstream checkpoint eviction removes min-step-adjacent entries first, then the oldest entry, then any entry with the same `n_tokens`. [source]
The log strings of the three paths are `erasing context checkpoint too close to an earlier one`, `erasing old context checkpoint` and `superseding context checkpoint at n_tokens`. [source]
The min-step proximity pass runs only when `checkpoints.size() + 1 >= n_ctx_checkpoints`, with the comment that otherwise short prompts would keep just the oldest checkpoint. [source]
A checkpoint created by the same task id is exempt from the proximity pass. [source]
A new checkpoint at an existing `n_tokens` replaces the old one instead of adding a duplicate. [source]
The master code still carries a TODO tag `[TAG_CHECKPOINTS_FIX_POS_MIN]` saying the saved checkpoint is wrongly assumed to cover `[pos_min, pos_max]`, untrue for SWA models (PR 24411 comment). [source]
PR 25592 listed neighbor-distance eviction in its first design and dropped it after PRs 25472 and 25649 landed overlapping work. [source]
PR 25592 exempts end-of-prompt checkpoints from `--checkpoint-min-step` in its first design because the next agent request diverges right at the end of the previous prompt. [source]
On a 38.5K-token prompt, creating a 230.431 MiB checkpoint took about 0.5 to 0.7 s and the restore cut a 40.8 s prefill to 0.1 to 0.3 s. [source]
A reporter ran `--ctx-checkpoints 128` with 149.626 MiB checkpoints, so a slot may hold about 18.7 GiB of checkpoint state at the limit. [source]
PR 25420 reads: "The server checkpoint logic breaks on every user message start, which does not respect --checkpoint-min-step and negatively impacts prefill speed." [source]
No upstream source read here defines a tokens-per-byte or hit-decay checkpoint score. [source]

Corrections and disagreements

Premise check, CONTRADICTS the concept name: no upstream file read here scores checkpoints by tokens per byte or decays a hit count. Upstream uses oldest-first plus min-step proximity. If such a scorer exists it is in a fork (the batch brief links it to CachyLlama), and no readable source documents it. [source]

Children

No children recorded.

← the whole tree · 3D view· how to read this page