MLX-LM perplexity and lm-evaluation
Parent: MLX-LM Python generation sampling and evaluation · Published reference · snapshot 2026-10-23 · viability 4.65 / 5 (research)
↓ Facts as markdownall context files
Source-anchored research for MLX-LM perplexity and lm-evaluation. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Structure and components
- The current `mlx-lm` package declares `lm-eval` and `tqdm` in the optional `evaluate` extra, so the harness dependency is not part of the base dependency list. (high confidence) (verified as of 2026-10-23) [source]
How it works
- `mlx_lm.lora --test` computes test-set perplexity using `test.jsonl` from the dataset named by `--data`. The `--mask-prompt` option changes the scored span from every token to the completion, so compare test perplexities only when both runs use the same masking policy. (high confidence) (verified as of 2026-10-23) [source]
- Perplexity measures next-token fit on the scored text; it does not by itself establish downstream task performance. MLX-LM exposes text perplexity and named benchmark tasks through separate commands, while research on benchmark leaderboards shows that task scores can shift under benchmark perturbations. Report both metrics when both text fit and task capability matter. (medium confidence) [source]
- Harness task results depend on task and prompt protocol: its CLI exposes task selection, few-shot count, chat-template use, generation arguments, and example limits, while an ACL 2024 study found that small changes such as multiple-choice option order or answer-selection method shifted MMLU leaderboard rankings by as many as eight places. Record the exact task version and scoring configuration when reporting a score. (high confidence) [source]
How-to and procedures
- Use --apply-chat-template or --no-apply-chat-template to explicitly enable or disable evaluation chat templating; argparse.BooleanOptionalAction provides both forms. --confirm-run-unsafe-code explicitly permits harness tasks that execute untrusted code and defaults to false. It is separate from --trust-remote-code for tokenizer loading. (low confidence) (verified as of 2026-10-23) [source]
Measurements and reference values
- A January 2026 MLX-LM issue reported that `mlx_lm.evaluate` failed while compiling a Metal kernel with `unknown type name 'bfloat16_t'` on `mlx-lm 0.30.4`, `mlx 0.30.4.dev`, and macOS 26.2. The reporter said deleting `build/` and rebuilding resolved that installation; the thread did not establish a general root cause or current status. (low confidence) [source]
Problems, failure modes and limitations
- As of the inspected source, `mlx_lm.perplexity` computes next-token cross-entropy, reports `exp(mean(token_loss))`, and estimates perplexity standard error with a delta approximation. Its CLI defaults are `allenai/tulu-3-sft-mixture`, 256 examples, sequence length 512, batch size 8, and seed 123; it randomly permutes dataset rows, concatenates tokenized contents, and reshapes the stream into fixed-length sequences. (high confidence) (verified as of 2026-10-23) [source]
- For repeatable comparisons, record model and tokenizer, corpus and split, sample count, context policy, random seed, tasks, few-shot count, prompt-template setting, limit, and package versions. These are reporting requirements, not a claim that every MLX-LM CLI exposes every control: the inspected standalone perplexity loader hardcodes its dataset splits and loads the tokenizer with the model. The project benchmark notes request MLX and MLX-LM versions, and the cited ACL study documents benchmark sensitivity. (medium confidence) (verified as of 2026-10-23) [source]
- A July 2025 import failure report traced `mlx_lm.evaluate` to access of `lm_eval.models.huggingface.HFLM.tokenizer_name`; the issue was closed as fixed in PR #275. Current `evaluate.py` instead imports `LM` and `register_model` from `lm_eval.api` and derives the tokenizer name from the loaded tokenizer, so that specific internal-attribute access is absent in the inspected source. This does not verify every current version combination. (medium confidence) (verified as of 2026-10-23) [source]
- A January 2024 upstream harness report recorded NaN perplexity and sharply lower LAMBADA accuracy for Pythia-70M with automatic dtype; setting `dtype=float32` restored the earlier reported score, and a maintainer noted small Pythia models were sensitive to 16-bit precision. This is a historical numeric-precision example, not evidence of a current MLX-LM failure. (low confidence) [source]
- mlx_lm.evaluate requires --model <model> and --tasks <task...>. --num-shots selects the few-shot example count; --limit caps examples per task; --output-dir selects the directory for result files and defaults to the current directory. These CLI options are forwarded to the harness or used for writing its results. (low confidence) (verified as of 2026-10-23) [source]
Comparisons and alternatives
- The harness uses `loglikelihood_rolling` for perplexity over a text distribution. MLX-LM scores documents separately. At revision 0d4d3d04, its docstring promises context-bounded windows with maximal context, but the implementation calls `_score_fn`, which processes default 2048-token steps with a continuing cache rather than constructing those windows. Compare implementations as well as documented context policies. (high confidence) (verified as of 2026-10-23) [source]
- Treat `--limit` runs as smoke tests rather than published benchmark scores: the harness CLI documents `--limit` as testing-only, and the MLX evaluation guide makes the same distinction for limited task runs. (high confidence) (verified as of 2026-10-23) [source]
- At revision 0d4d3d04, standalone `mlx_lm.perplexity` concatenates tokenized dataset rows and reshapes them into disjoint fixed-length sequences. `MLXLM.loglikelihood_rolling` scores documents separately; `_score_fn` shifts encoded inputs and next-token targets, so the first encoded token serves as context rather than a scored target. Its identity depends on tokenizer special-token and template settings. The rolling method does not explicitly prepend EOT, despite its docstring example. Align these protocols before comparing perplexity. (high confidence) [source]
Facts and statements
- `mlx_lm.evaluate` registers the `mlxlm` backend with lm-evaluation-harness and implements `loglikelihood` and `loglikelihood_rolling`. The current `mlx-lm[evaluate]` extra installs `lm-eval` and `tqdm`, and the package registers the `mlx_lm.evaluate` command entry point. (high confidence) (verified as of 2026-10-23) [source]
- The standalone `mlx_lm.perplexity` CLI has no `--adapter-path` option in its current parser. To score an adapter, use `mlx_lm.lora --test` on the held-out dataset or fuse the adapter before using the standalone perplexity command. (high confidence) (verified as of 2026-10-23) [source]
- Token-level perplexity depends on tokenization. A 2026 EACL study reports word-level perplexity through lm-evaluation-harness to enable cross-tokenizer comparisons; a 2025 NeurIPS study cautions that vocabulary-size and likelihood-bound differences made validation perplexity an unsuitable effectiveness comparison in its diffusion-language-model setting. For cross-tokenizer comparisons, state the normalization and pair perplexity with task results. (medium confidence) [source]
- The current MLX-LM `BENCHMARKS.md` example spells the task option `--task`, while `mlx_lm.evaluate` declares the required option as `--tasks`. Use the plural spelling declared by the parser; whether argparse accepts the singular spelling as an abbreviation was not verified. (medium confidence) (verified as of 2026-10-23) [source]
- `mlx_lm.evaluate` registers an MLX model with EleutherAI's `lm-evaluation-harness` and implements continuation log-likelihood, rolling log-likelihood, and generation. The harness model guide defines rolling log-likelihood as the interface used to evaluate perplexity on a data distribution. (high confidence) (verified as of 2026-10-23) [source]
- Perplexity depends on tokenization and context-window protocol: Hugging Face's documentation says tokenization directly affects PPL and demonstrates a score change between disjoint windows and a strided sliding window. Compare PPL only after matching tokenizer, corpus preprocessing, scored tokens, and context/scoring protocol. (high confidence) [source]
- Adapter test evaluation computes test_ppl=math.exp(test_loss). Run mlx_lm.lora --model <base> --adapter-path <adapter_directory> --data <dataset_directory> --test; with --test and no --train, the entry point loads that adapter before evaluating test.jsonl. An empty --adapter-path explicitly skips adapter loading for base-model testing. (low confidence) (verified as of 2026-10-23) [source]
Children
- No children recorded.