# MLX-LM Python generation and streaming contracts > Source-anchored research for MLX-LM Python generation and streaming contracts. Run status: COMPLETED. Parent: [MLX-LM Python generation sampling and evaluation](https://llms-explorer.com/tree/mlx-lm-python-generation-sampling-and-evaluation/) · 3 facets · 16 facts · page: https://llms-explorer.com/tree/mlx-lm-python-generation-and-streaming-contracts/ ## How it works - `stream_generate` is a generator of `GenerationResponse` objects, not strings. Callers read each response's `.text`; that field may be empty. Ordinary per-token responses have `finish_reason=None`, and the final response reports `stop` for EOS or `length` when generation ends at the token cap. (high confidence) (verified as of 2026-10-23) — [source](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/generate.py) - The current implementation creates its default generation stream with `mx.new_thread_local_stream`. A third-party reproduction on MLX-LM 0.31.3 reported `RuntimeError: There is no Stream(gpu, N) in current thread` when a sliding-window model was loaded on one thread and generated on another; keep load and generation on the same thread for affected model and runtime combinations until verified locally. (medium confidence) (verified as of 2026-10-23) — [source](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/generate.py) - generate is the whole-response helper: it iterates stream_generate, concatenates each response.text segment, and returns the generated continuation when text exists; verbose=True prints the generated text and final token, speed, and memory statistics. (high confidence) (verified as of 2026-10-23) — [source](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/generate.py) - The inspected `generate` function is annotated to return `str`, but its empty-output path returns `None` when `verbose=True`. With the default `verbose=False`, the same empty-output case returns an empty string. This control-flow distinction requires indentation-preserving source evidence. (medium confidence) (verified as of 2026-10-23) — [source](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/generate.py) ## Problems, failure modes and limitations - In current upstream main, stream_generate yields intermediate responses with finish_reason=None, then finalizes the detokenizer and yields a terminal response marked stop for EOS or length when the token limit ends generation. If EOS ends generation, the terminal response's token field still contains the EOS token, although that token is not appended to decoded text. (low confidence) (verified as of 2026-10-23) — [source](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/generate.py) - Current upstream main rejects stream_generate(max_tokens=0) with ValueError and directs callers to use -1 for no token limit. (low confidence) (verified as of 2026-10-23) — [source](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/generate.py) ## Facts and statements - At revision 0d4d3d04, `generate(model, tokenizer, prompt, ...)` concatenates streamed `.text` values and normally returns a Python string. If the assembled text is empty, `verbose=True` prints `No text generated for this prompt` and returns `None`; the default `verbose=False` path returns the empty string. (medium confidence) (verified as of 2026-10-23) — [source](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/generate.py) - Advancing `stream_generate(..., max_tokens=0)` raises `ValueError` in the current `main` implementation. A third-party guide still says this path raises `UnboundLocalError`; that note is stale relative to the current source. (medium confidence) (verified as of 2026-10-23) — [source](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/generate.py) - The documented `stream_generate` prompt types are `str`, `mx.array`, and `List[int]`. A Transformers `BatchEncoding` returned by `apply_chat_template` can violate that input shape: an MLX-LM issue traced a server crash on Transformers 5.0+ to a chat-template call that omitted `return_dict=False`. (medium confidence) (verified as of 2026-10-23) — [source](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/generate.py) - A supplied `prompt_cache` is updated in place. An open MLX-LM issue still reports that direct calls with extending prompts lack first-class prefix-aware reuse, so callers using that pattern must track which tokens the cache represents and trim it themselves. (medium confidence) (verified as of 2026-10-23) — [source](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/generate.py) - The current mlx_lm package exports load, generate, and stream_generate at package top level; load returns the (model, tokenizer) pair used by the generation helpers. (high confidence) (verified as of 2026-10-23) — [source](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/__init__.py) - stream_generate returns a Python generator and accepts a prompt as str, mx.array, or List[int]; it tokenizes string prompts internally and yields GenerationResponse objects. (high confidence) (verified as of 2026-10-23) — [source](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/generate.py) - GenerationResponse.text is the next decoded text segment and may be empty; each response also carries the token, log probabilities, draft-model flag, prompt and generation token counts and rates, peak memory, and optional finish_reason. (high confidence) (verified as of 2026-10-23) — [source](https://github.com/ml-explore/mlx-lm/blob/main/mlx_lm/generate.py) - Pass revision="" to load(path_or_hf_repo, revision=...) to select the Hugging Face model revision. The loader forwards revision to its download helper; pin an immutable commit for reproducible remote model selection. (low confidence) (verified as of 2026-10-23) — [source](https://raw.githubusercontent.com/ml-explore/mlx-lm/0d4d3d04f8ab7ce4a51e7209e8615755a48ba94d/mlx_lm/utils.py) - With an existing prompt_cache, generate_step forwards the supplied prompt tokens to the model with that cache; it does not detect an already cached prefix or strip duplicate tokens. Callers must pass only the uncached suffix and ensure that the cache matches the preceding token history. The supplied cache is mutated in place. (low confidence) (verified as of 2026-10-23) — [source](https://raw.githubusercontent.com/ml-explore/mlx-lm/0d4d3d04f8ab7ce4a51e7209e8615755a48ba94d/mlx_lm/generate.py) - stream_generate requires a draft_model to use the same tokenizer as the main model. The native CLI checks draft_tokenizer.vocab_size against tokenizer.vocab_size and raises ValueError("Draft model tokenizer does not match model tokenizer.") on a mismatch; equal vocabulary sizes alone do not prove identical token mappings. (low confidence) (verified as of 2026-10-23) — [source](https://raw.githubusercontent.com/ml-explore/mlx-lm/0d4d3d04f8ab7ce4a51e7209e8615755a48ba94d/mlx_lm/generate.py)