# MLX-LM stateful logits processor rollback during speculative decoding > Source-anchored research for MLX-LM stateful logits processor rollback during speculative decoding. Run status: COMPLETED. Parent: [MLX-LM native multimodal generation and request lifecycle](https://llms-explorer.com/tree/mlx-lm-native-multimodal-generation-and-request-lifecycle/) · 3 facets · 13 facts · page: https://llms-explorer.com/tree/mlx-lm-stateful-logits-processor-rollback-during-speculative-decoding/ ## Structure and components - Historical issue #1156 names mtp_generate_step and proposes a lazy-evaluation explanation that a maintainer disputes. The reviewed generate.py at 0d4d3d04f8ab defines speculative_generate_step and contains no mtp_generate_step definition. Keep the historical report separate from this pinned path; do not treat its proposed fix as evidence that the pinned speculative path implements generic processor-state rollback. (medium confidence) (verified as of 2026-10-23) — [source](https://raw.githubusercontent.com/ml-explore/mlx-lm/0d4d3d04f8ab7ce4a51e7209e8615755a48ba94d/mlx_lm/generate.py) ## How it works - LM Studio's mlx-engine creates an Outlines JSONLogitsProcessor when json_schema is set and passes it to MLX-LM stream_generate alongside draft_model; thus structured decoding can enter the same speculative generation interface where processor-state rollback matters. (medium confidence) (verified as of 2026-10-23) — [source](https://github.com/lmstudio-ai/mlx-engine/blob/main/mlx_engine/generate.py) - A downstream MLX MTP integration documented routing constrained requests to BatchGenerator when its speculative path could not preserve stateful grammar, budget, and penalty processors. This is a fallback option when the speculative loop lacks processor state and rollback support. (medium confidence) (verified as of 2026-10-05) — [source](https://github.com/jundot/omlx/issues/2399) - A caller-owned LogitsProcessor becomes stateful when its object or closure mutates an FSM, grammar matcher, or counter across calls. Its callable contract takes (tokens, logits) and returns revised logits; it has no native state object or snapshot argument. Because speculative code invokes it before acceptance is known, per-call mutation can outlive a rejected candidate. (medium confidence) (verified as of 2026-10-23) — [source](https://raw.githubusercontent.com/ml-explore/mlx-lm/0d4d3d04f8ab7ce4a51e7209e8615755a48ba94d/mlx_lm/generate.py) - After acceptance/rejection, speculative_generate_step truncates prev_tokens by max(num_draft-n,1) positions and calls _rewind_cache(num_draft,n). That helper trims model_cache by num_draft-n and draft_cache by max(num_draft-n-1,0); finally calls it again when leaving the loop. This restores native token/cache bookkeeping but invokes no snapshot/restore callback on caller-owned processor state. (low confidence) (verified as of 2026-10-23) — [source](https://raw.githubusercontent.com/ml-explore/mlx-lm/0d4d3d04f8ab7ce4a51e7209e8615755a48ba94d/mlx_lm/generate.py) - Processor integration guidance: reconstruct derived FSM/counter state from the supplied token history on each call, including after a shorter history replaces a provisional one; do not assume calls advance an accepted prefix monotonically. A mutable external FSM needs a reset-and-replay or equivalent snapshot/rollback capability for rejected candidates, coordinated by its integration because the native callable interface supplies no such lifecycle callbacks. The upstream thinking-budget discussion explicitly contrasts stateless recomputation with a proposed save_state()/restore_state() protocol. (medium confidence) (verified as of 2026-10-23) — [source](https://raw.githubusercontent.com/ml-explore/mlx-lm/0d4d3d04f8ab7ce4a51e7209e8615755a48ba94d/mlx_lm/generate.py) ## Facts and statements - In MTP speculative decoding, a stateful processor can mutate on a candidate token that is later rejected; if its state is not restored or rebuilt from the accepted prefix, grammar or counter state can drift from the generated history. (high confidence) — [source](https://github.com/ml-explore/mlx-lm/pull/1196) - MLX-LM's MTP work corrected stale prev_tokens handling and rejected-draft history separately from processor state. The thinking-budget processor can still advance during verification for a bonus token that is later rejected; the maintainer says recomputing state from accepted token history or adding an explicit save_state()/restore_state() protocol would prevent counter drift. (high confidence) (verified as of 2026-10-23) — [source](https://github.com/ml-explore/mlx-lm/issues/1175) - The current mlx-lm main speculative_generate_step calls logits processors during draft and verification _step calls, then rejection cleanup truncates prev_tokens and trims model caches; the public LogitsProcessor alias is only Callable[[mx.array, mx.array], mx.array], so the interface exposes no generic processor-state rollback hook. (high confidence) (verified as of 2026-10-23) — [source](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/generate.py) - Two remedies are described for stateful processors: recompute state from the accepted token history on each call, or add processor state snapshot and restore around speculative verification; the latter is a proposed interface need, not a generic method in the reviewed MLX-LM callable API. (medium confidence) (verified as of 2026-10-23) — [source](https://github.com/ml-explore/mlx-lm/pull/1196) - Independent MTP guidance also treats this as accepted-prefix bookkeeping: Metrale says sampler state must track accepted tokens and rejected drafts must be unwound, while an OMLX MTP discussion says a grammar matcher must advance on accepted tokens and roll back on rejected drafts. (high confidence) — [source](https://docs.metrale.ai/deep-dives/mtp) - In speculative_generate_step at 0d4d3d04f8ab, _draft_generate calls _step(draft_model, draft_cache, y) for each proposal. _step appends y to prev_tokens and invokes _process_and_sample(prev_tokens, logits), which applies processors in list order before logsumexp normalization and sampling. Thus processors run while proposals are being formed, not only for accepted output tokens. (low confidence) (verified as of 2026-10-23) — [source](https://raw.githubusercontent.com/ml-explore/mlx-lm/0d4d3d04f8ab7ce4a51e7209e8615755a48ba94d/mlx_lm/generate.py) - Before main-model verification, speculative_generate_step truncates prev_tokens to remove the draft-side context extension and calls _step(model, model_cache, concatenated_input_and_drafts, num_draft+1). For n_predict>1, _step first shortens y by n_predict-1 positions; on each prediction it appends the current y to prev_tokens, calls processors with that history and logits[:, i, :], and uses the sampled main-model y for the next prediction. These are the pinned implementation inputs, not a guarantee that every processor call corresponds to an accepted token. (low confidence) (verified as of 2026-10-23) — [source](https://raw.githubusercontent.com/ml-explore/mlx-lm/0d4d3d04f8ab7ce4a51e7209e8615755a48ba94d/mlx_lm/generate.py)