MLX-LM Python generation sampling and evaluation

Parent: MLX and mlx-lm on Apple silicon · Published reference · snapshot 2026-10-23

Source-anchored research for MLX-LM Python generation sampling and evaluation. Run status: COMPLETED.

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Definitions

apply_top_k rejects top_k unless it is an integer strictly greater than zero and strictly less than logprobs.shape[-1] (the vocabulary size), raising ValueError otherwise. make_sampler separately treats top_k=0 as disabling its top-k filter. (low confidence) (verified as of 2026-10-23) [source]

Structure and components

The current `mlx-lm` package declares `lm-eval` and `tqdm` in the optional `evaluate` extra, so the harness dependency is not part of the base dependency list. (high confidence) (verified as of 2026-10-23) [source]

How it works

make_sampler returns greedy argmax immediately when temp == 0, so the filtering chain is bypassed at zero temperature. At nonzero temperature, the factory applies enabled top-p, min-p, XTC, then top-k filters to log probabilities before categorical sampling. (high confidence) (verified as of 2026-10-23) [source]
A report against mlx-lm 0.31.3 described TypeError: 'NoneType' object is not iterable when continuous batches mixed requests with and without logits processors. Upstream PR #1826 merged on 2026-09-04, normalizing per-sequence samplers and processors and closing the report. (high confidence) (verified as of 2026-10-23) [source]
`mlx_lm.lora --test` computes test-set perplexity using `test.jsonl` from the dataset named by `--data`. The `--mask-prompt` option changes the scored span from every token to the completion, so compare test perplexities only when both runs use the same masking policy. (high confidence) (verified as of 2026-10-23) [source]
Perplexity measures next-token fit on the scored text; it does not by itself establish downstream task performance. MLX-LM exposes text perplexity and named benchmark tasks through separate commands, while research on benchmark leaderboards shows that task scores can shift under benchmark perturbations. Report both metrics when both text fit and task capability matter. (medium confidence) [source]
Harness task results depend on task and prompt protocol: its CLI exposes task selection, few-shot count, chat-template use, generation arguments, and example limits, while an ACL 2024 study found that small changes such as multiple-choice option order or answer-selection method shifted MMLU leaderboard rankings by as many as eight places. Record the exact task version and scoring configuration when reporting a score. (high confidence) [source]
`stream_generate` is a generator of `GenerationResponse` objects, not strings. Callers read each response's `.text`; that field may be empty. Ordinary per-token responses have `finish_reason=None`, and the final response reports `stop` for EOS or `length` when generation ends at the token cap. (high confidence) (verified as of 2026-10-23) [source]
The current implementation creates its default generation stream with `mx.new_thread_local_stream`. A third-party reproduction on MLX-LM 0.31.3 reported `RuntimeError: There is no Stream(gpu, N) in current thread` when a sliding-window model was loaded on one thread and generated on another; keep load and generation on the same thread for affected model and runtime combinations until verified locally. (medium confidence) (verified as of 2026-10-23) [source]
generate is the whole-response helper: it iterates stream_generate, concatenates each response.text segment, and returns the generated continuation when text exists; verbose=True prints the generated text and final token, speed, and memory statistics. (high confidence) (verified as of 2026-10-23) [source]
The inspected `generate` function is annotated to return `str`, but its empty-output path returns `None` when `verbose=True`. With the default `verbose=False`, the same empty-output case returns an empty string. This control-flow distinction requires indentation-preserving source evidence. (medium confidence) (verified as of 2026-10-23) [source]

How-to and procedures

Use --apply-chat-template or --no-apply-chat-template to explicitly enable or disable evaluation chat templating; argparse.BooleanOptionalAction provides both forms. --confirm-run-unsafe-code explicitly permits harness tasks that execute untrusted code and defaults to false. It is separate from --trust-remote-code for tokenizer loading. (low confidence) (verified as of 2026-10-23) [source]

Measurements and reference values

A January 2026 MLX-LM issue reported that `mlx_lm.evaluate` failed while compiling a Metal kernel with `unknown type name 'bfloat16_t'` on `mlx-lm 0.30.4`, `mlx 0.30.4.dev`, and macOS 26.2. The reporter said deleting `build/` and rebuilding resolved that installation; the thread did not establish a general root cause or current status. (low confidence) [source]

Problems, failure modes and limitations

As of the inspected source, `mlx_lm.perplexity` computes next-token cross-entropy, reports `exp(mean(token_loss))`, and estimates perplexity standard error with a delta approximation. Its CLI defaults are `allenai/tulu-3-sft-mixture`, 256 examples, sequence length 512, batch size 8, and seed 123; it randomly permutes dataset rows, concatenates tokenized contents, and reshapes the stream into fixed-length sequences. (high confidence) (verified as of 2026-10-23) [source]
For repeatable comparisons, record model and tokenizer, corpus and split, sample count, context policy, random seed, tasks, few-shot count, prompt-template setting, limit, and package versions. These are reporting requirements, not a claim that every MLX-LM CLI exposes every control: the inspected standalone perplexity loader hardcodes its dataset splits and loads the tokenizer with the model. The project benchmark notes request MLX and MLX-LM versions, and the cited ACL study documents benchmark sensitivity. (medium confidence) (verified as of 2026-10-23) [source]
A July 2025 import failure report traced `mlx_lm.evaluate` to access of `lm_eval.models.huggingface.HFLM.tokenizer_name`; the issue was closed as fixed in PR #275. Current `evaluate.py` instead imports `LM` and `register_model` from `lm_eval.api` and derives the tokenizer name from the loaded tokenizer, so that specific internal-attribute access is absent in the inspected source. This does not verify every current version combination. (medium confidence) (verified as of 2026-10-23) [source]
A January 2024 upstream harness report recorded NaN perplexity and sharply lower LAMBADA accuracy for Pythia-70M with automatic dtype; setting `dtype=float32` restored the earlier reported score, and a maintainer noted small Pythia models were sensitive to 16-bit precision. This is a historical numeric-precision example, not evidence of a current MLX-LM failure. (low confidence) [source]
mlx_lm.evaluate requires --model <model> and --tasks <task...>. --num-shots selects the few-shot example count; --limit caps examples per task; --output-dir selects the directory for result files and defaults to the current directory. These CLI options are forwarded to the harness or used for writing its results. (low confidence) (verified as of 2026-10-23) [source]
In current upstream main, stream_generate yields intermediate responses with finish_reason=None, then finalizes the detokenizer and yields a terminal response marked stop for EOS or length when the token limit ends generation. If EOS ends generation, the terminal response's token field still contains the EOS token, although that token is not appended to decoded text. (low confidence) (verified as of 2026-10-23) [source]
Current upstream main rejects stream_generate(max_tokens=0) with ValueError and directs callers to use -1 for no token limit. (low confidence) (verified as of 2026-10-23) [source]

Comparisons and alternatives

The harness uses `loglikelihood_rolling` for perplexity over a text distribution. MLX-LM scores documents separately. At revision 0d4d3d04, its docstring promises context-bounded windows with maximal context, but the implementation calls `_score_fn`, which processes default 2048-token steps with a continuing cache rather than constructing those windows. Compare implementations as well as documented context policies. (high confidence) (verified as of 2026-10-23) [source]
Treat `--limit` runs as smoke tests rather than published benchmark scores: the harness CLI documents `--limit` as testing-only, and the MLX evaluation guide makes the same distinction for limited task runs. (high confidence) (verified as of 2026-10-23) [source]
At revision 0d4d3d04, standalone `mlx_lm.perplexity` concatenates tokenized dataset rows and reshapes them into disjoint fixed-length sequences. `MLXLM.loglikelihood_rolling` scores documents separately; `_score_fn` shifts encoded inputs and next-token targets, so the first encoded token serves as context rather than a scored target. Its identity depends on tokenizer special-token and template settings. The rolling method does not explicitly prepend EOT, despite its docstring example. Align these protocols before comparing perplexity. (high confidence) [source]

Facts and statements

MLX-LM's Python generate and stream_generate functions accept separate sampler and logits_processors callables. Each processor receives token history and current logits; processors run in list order before normalization, and current generate.py passes normalized log-probabilities to the sampler. (high confidence) (verified as of 2026-10-23) [source]
sample_utils.make_sampler returns greedy argmax for temp=0; otherwise it applies enabled top-p, min-p, XTC, and top-k filters before categorical sampling at the requested temperature. make_logits_processors builds optional additive logit bias followed by enabled repetition, presence, and frequency penalties. The independent csm-mlx README demonstrates make_sampler(temp=0.8, top_k=50) and logit-bias use. (high confidence) (verified as of 2026-10-23) [source]
The independent Exo MLX wrapper composes an application-defined ban_token_ids(eos_ids) processor ahead of built-in penalties and passes the processor list per sequence to MLX-LM's BatchGenerator.insert, providing a concrete pattern for combining custom filters with make_logits_processors in a batched path. (high confidence) (verified as of 2026-10-23) [source]
April 2026 reports for mlx-lm 0.31.2 describe stateful processors seeing stale token history in stream or MTP paths: one report attributes the observation to lazy token arrays, and another says MTP draft calls omitted the backbone's just-sampled token. The MTP report author later described a vendored fix that snapshots token context and restores it after a rejected draft; the earlier report says stateless repetition penalties were unaffected. Verify this version-specific behavior against the release in use. (medium confidence) (verified as of 2026-04-28) [source]
The current generate_step API accepts optional sampler and logits_processors arguments, and stream_generate forwards its extra keyword arguments to generate_step. Each processor receives (tokens, logits) and returns revised logits; generation applies processors before normalizing logits and sampling. This lets callers pass custom callables as well as the supplied factories. (high confidence) (verified as of 2026-10-23) [source]
make_logits_processors can build processors for additive logit_bias, sign-aware repetition penalty, presence penalty, and frequency penalty. Its three penalty context-size defaults are 20 tokens; presence subtracts once for tokens in the context, while frequency subtracts per occurrence. (high confidence) (verified as of 2026-10-23) [source]
Independent consumers demonstrate the intended integration pattern: csm-mlx passes MLX-LM's sampler to its generator and documents logit bias through logits_processors; Exo's MLX backend imports both factories and builds sampler and processor callables for batched generation. (high confidence) (verified as of 2026-10-23) [source]
A Xinference issue opened on 2025-04-09 records that importing make_logits_processors and make_sampler from mlx_lm.utils failed; a follow-up shows the integration importing them from mlx_lm.sample_utils. Current MLX-LM source defines the factories in sample_utils.py, so integrations should check the installed package's module path. (medium confidence) (verified as of 2026-10-23) [source]
An open MLX-LM issue reports that mtp_generate_step in mlx-lm 0.31.2 passed stale prev_tokens to stateful processors on speculative draft calls. The author says the current main MTP path had since been restructured, so this version-specific report does not establish a defect in ordinary serial generate_step. (low confidence) (verified as of 2026-10-23) [source]
A Sampler callable receives log-probabilities and returns token indices as an MLX array. In generate_step, logits processors first modify raw logits in list order; the code then subtracts mx.logsumexp(logits, keepdims=True) and passes the resulting log-probabilities to sampler. (high confidence) (verified as of 2026-10-23) [source]
min_p removes candidates whose probability is below min_p times the most probable token probability. top_p instead filters by cumulative probability mass. min_tokens_to_keep protects a minimum number of the highest-probability candidates from min_p filtering and must be a positive integer. (low confidence) (verified as of 2026-10-23) [source]
make_logits_processors takes logit_bias as a dictionary mapping integer token IDs to additive bias values. It constructs index and value arrays from the dictionary keys and values, then adds those values to the corresponding logits. (low confidence) (verified as of 2026-10-23) [source]
For tokens in its selected context, repetition_penalty multiplies negative logits by the penalty and divides nonnegative logits by the penalty. presence_penalty subtracts its penalty once for a token present in context, while frequency_penalty subtracts a penalty scaled by that token's occurrence count. (low confidence) (verified as of 2026-10-23) [source]
`mlx_lm.evaluate` registers the `mlxlm` backend with lm-evaluation-harness and implements `loglikelihood` and `loglikelihood_rolling`. The current `mlx-lm[evaluate]` extra installs `lm-eval` and `tqdm`, and the package registers the `mlx_lm.evaluate` command entry point. (high confidence) (verified as of 2026-10-23) [source]
The standalone `mlx_lm.perplexity` CLI has no `--adapter-path` option in its current parser. To score an adapter, use `mlx_lm.lora --test` on the held-out dataset or fuse the adapter before using the standalone perplexity command. (high confidence) (verified as of 2026-10-23) [source]
Token-level perplexity depends on tokenization. A 2026 EACL study reports word-level perplexity through lm-evaluation-harness to enable cross-tokenizer comparisons; a 2025 NeurIPS study cautions that vocabulary-size and likelihood-bound differences made validation perplexity an unsuitable effectiveness comparison in its diffusion-language-model setting. For cross-tokenizer comparisons, state the normalization and pair perplexity with task results. (medium confidence) [source]
The current MLX-LM `BENCHMARKS.md` example spells the task option `--task`, while `mlx_lm.evaluate` declares the required option as `--tasks`. Use the plural spelling declared by the parser; whether argparse accepts the singular spelling as an abbreviation was not verified. (medium confidence) (verified as of 2026-10-23) [source]
`mlx_lm.evaluate` registers an MLX model with EleutherAI's `lm-evaluation-harness` and implements continuation log-likelihood, rolling log-likelihood, and generation. The harness model guide defines rolling log-likelihood as the interface used to evaluate perplexity on a data distribution. (high confidence) (verified as of 2026-10-23) [source]
Perplexity depends on tokenization and context-window protocol: Hugging Face's documentation says tokenization directly affects PPL and demonstrates a score change between disjoint windows and a strided sliding window. Compare PPL only after matching tokenizer, corpus preprocessing, scored tokens, and context/scoring protocol. (high confidence) [source]
Adapter test evaluation computes test_ppl=math.exp(test_loss). Run mlx_lm.lora --model <base> --adapter-path <adapter_directory> --data <dataset_directory> --test; with --test and no --train, the entry point loads that adapter before evaluating test.jsonl. An empty --adapter-path explicitly skips adapter loading for base-model testing. (low confidence) (verified as of 2026-10-23) [source]
At revision 0d4d3d04, `generate(model, tokenizer, prompt, ...)` concatenates streamed `.text` values and normally returns a Python string. If the assembled text is empty, `verbose=True` prints `No text generated for this prompt` and returns `None`; the default `verbose=False` path returns the empty string. (medium confidence) (verified as of 2026-10-23) [source]
Advancing `stream_generate(..., max_tokens=0)` raises `ValueError` in the current `main` implementation. A third-party guide still says this path raises `UnboundLocalError`; that note is stale relative to the current source. (medium confidence) (verified as of 2026-10-23) [source]
The documented `stream_generate` prompt types are `str`, `mx.array`, and `List[int]`. A Transformers `BatchEncoding` returned by `apply_chat_template` can violate that input shape: an MLX-LM issue traced a server crash on Transformers 5.0+ to a chat-template call that omitted `return_dict=False`. (medium confidence) (verified as of 2026-10-23) [source]
A supplied `prompt_cache` is updated in place. An open MLX-LM issue still reports that direct calls with extending prompts lack first-class prefix-aware reuse, so callers using that pattern must track which tokens the cache represents and trim it themselves. (medium confidence) (verified as of 2026-10-23) [source]
The current mlx_lm package exports load, generate, and stream_generate at package top level; load returns the (model, tokenizer) pair used by the generation helpers. (high confidence) (verified as of 2026-10-23) [source]
stream_generate returns a Python generator and accepts a prompt as str, mx.array, or List[int]; it tokenizes string prompts internally and yields GenerationResponse objects. (high confidence) (verified as of 2026-10-23) [source]
GenerationResponse.text is the next decoded text segment and may be empty; each response also carries the token, log probabilities, draft-model flag, prompt and generation token counts and rates, peak memory, and optional finish_reason. (high confidence) (verified as of 2026-10-23) [source]
Pass revision="<branch, tag, or commit hash>" to load(path_or_hf_repo, revision=...) to select the Hugging Face model revision. The loader forwards revision to its download helper; pin an immutable commit for reproducible remote model selection. (low confidence) (verified as of 2026-10-23) [source]
With an existing prompt_cache, generate_step forwards the supplied prompt tokens to the model with that cache; it does not detect an already cached prefix or strip duplicate tokens. Callers must pass only the uncached suffix and ensure that the cache matches the preceding token history. The supplied cache is mutated in place. (low confidence) (verified as of 2026-10-23) [source]
stream_generate requires a draft_model to use the same tokenizer as the main model. The native CLI checks draft_tokenizer.vocab_size against tokenizer.vocab_size and raises ValueError("Draft model tokenizer does not match model tokenizer.") on a mismatch; equal vocabulary sizes alone do not prove identical token mappings. (low confidence) (verified as of 2026-10-23) [source]

Children

Stateful logits processor state rollback for speculative decoding (frontier)
Random sampling behavior across MLX worker threads (frontier)
MLX-LM custom sampling and logits processors
MLX-LM perplexity and lm-evaluation
MLX-LM Python generation and streaming contracts

← the whole tree · 3D view· how to read this page