MLX-LM native multimodal generation and request lifecycle

Parent: MLX and mlx-lm on Apple silicon · Published reference · snapshot 2026-10-23

Source-anchored research for MLX-LM native multimodal generation and request lifecycle. Run status: COMPLETED.

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Structure and components

In the built-in HTTP server's chat-template path, image content parts marked image_url, input_image, or image are accepted when the loaded model has a processor. The server fetches HTTP(S) images into data URLs, decodes RGB pixels, applies the processor to the templated message, and supplies token IDs plus image embeddings to generation. This includes the common Chat Completions image_url form, which may carry a URL or base64 image data. (high confidence) (verified as of 2026-10-23) [source]
In `mlx_lm.server` chat handling, image content is recognized on the chat-template path when a model processor is loaded; the server preserves parts with type `image_url`, `input_image`, or `image`. OpenAI's Chat Completions reference independently defines user image inputs as content-part arrays with an `image_url.url` field. (high confidence) (verified as of 2026-10-23) [source]
load_processor(model,config) returns None without a truthy vision_config. Otherwise it chooses config.processor_type or the final component of type(model).__module__, looks that name up in PROCESSORS, and invokes processor.from_config(config) when registered. At 0d4d3d04f8ab, PROCESSORS registers only "muse_spark": muse_spark.Processor; this does not establish support for arbitrary multimodal architectures. (low confidence) (verified as of 2026-10-23) [source]
A standard chat image part is {"type":"image_url","image_url":{"url":"<http(s) or data URL>"}} within a messages content list. _image_url extracts its URL; _inline_image_urls fetches remote bytes and replaces them with a base64 data URL before tokenization. _decode_image accepts that data URL, base64-decodes it, opens the bytes with Pillow, converts to RGB, and returns a uint8 NumPy array. (low confidence) (verified as of 2026-10-23) [source]
Historical issue #1156 names mtp_generate_step and proposes a lazy-evaluation explanation that a maintainer disputes. The reviewed generate.py at 0d4d3d04f8ab defines speculative_generate_step and contains no mtp_generate_step definition. Keep the historical report separate from this pinned path; do not treat its proposed fix as evidence that the pinned speculative path implements generic processor-state rollback. (medium confidence) (verified as of 2026-10-23) [source]

How it works

The current mlx_lm.server keeps an LRUPromptCache on its ResponseGenerator and inserts extracted prompt-segment KV caches into it. An issue report describes BrokenPipeError on disconnected responses while request sequences remained in the cache, but the reporter says they could not establish whether disconnections caused or accelerated the deadlock; a disconnect should not be assumed to free cached KV or be treated as a proven root cause. (medium confidence) (verified as of 2026-10-23) [source]
At pinned 0d4d3d04f8ab, APIHandler.handle_completion observes connection failures when wfile.write or flush raises during keepalive, SSE, or final response output; it does not poll for disconnects. ResponseGenerator.generate first returns ctx and a lazy _inner response iterator. Only consumption of that iterator invokes keepalive_callback on queued progress tuples. handle_completion consumes it inside the response try block, whose finally calls ctx.stop(), including when a keepalive or response write fails there. (low confidence) (verified as of 2026-10-23) [source]
When the server finds a nearest prompt-cache prefix, it counts image starts before the cached prefix and passes only the remaining `pixel_values` through `model.embed_inputs`; generation then uses those embeddings with the KV cache. The server appends generated token IDs to the cache key before storing the updated cache. MLX-VLM separately documents that a repeated-image feature-cache hit skips its vision encoder. (medium confidence) (verified as of 2026-10-23) [source]
LM Studio's mlx-engine creates an Outlines JSONLogitsProcessor when json_schema is set and passes it to MLX-LM stream_generate alongside draft_model; thus structured decoding can enter the same speculative generation interface where processor-state rollback matters. (medium confidence) (verified as of 2026-10-23) [source]
A downstream MLX MTP integration documented routing constrained requests to BatchGenerator when its speculative path could not preserve stateful grammar, budget, and penalty processors. This is a fallback option when the speculative loop lacks processor state and rollback support. (medium confidence) (verified as of 2026-10-05) [source]
A caller-owned LogitsProcessor becomes stateful when its object or closure mutates an FSM, grammar matcher, or counter across calls. Its callable contract takes (tokens, logits) and returns revised logits; it has no native state object or snapshot argument. Because speculative code invokes it before acceptance is known, per-call mutation can outlive a rejected candidate. (medium confidence) (verified as of 2026-10-23) [source]
After acceptance/rejection, speculative_generate_step truncates prev_tokens by max(num_draft-n,1) positions and calls _rewind_cache(num_draft,n). That helper trims model_cache by num_draft-n and draft_cache by max(num_draft-n-1,0); finally calls it again when leaving the loop. This restores native token/cache bookkeeping but invokes no snapshot/restore callback on caller-owned processor state. (low confidence) (verified as of 2026-10-23) [source]
Processor integration guidance: reconstruct derived FSM/counter state from the supplied token history on each call, including after a shorter history replaces a provisional one; do not assume calls advance an accepted prefix monotonically. A mutable external FSM needs a reset-and-replay or equivalent snapshot/rollback capability for rejected candidates, coordinated by its integration because the native callable interface supplies no such lifecycle callbacks. The upstream thinking-budget discussion explicitly contrasts stateless recomputation with a proposed save_state()/restore_state() protocol. (medium confidence) (verified as of 2026-10-23) [source]

How-to and procedures

To distinguish images that expand to the same image-token pattern, `mlx_lm.server` replaces each contiguous image-token run in the prompt-cache key with a negative integer derived from the first 15 hexadecimal digits of SHA-256 over the inlined image data URL. MLX-VLM documents a separate `VisionFeatureCache` that keys projected vision features by image path. (medium confidence) (verified as of 2026-10-23) [source]

Problems, failure modes and limitations

stream_generate enters wired_limit around generation. When maybe_set_recommended_wired_limit returns a prior limit, wired_limit synchronizes the supplied streams (or the default synchronization path) and restores that limit in finally. If the helper returns None, wired_limit yields and returns without that synchronization or restoration. Closing or exhausting an entered generator unwinds its context; LM Studio's sequential adapter explicitly closes its wrapped stream in finally. (high confidence) (verified as of 2026-10-23) [source]

Comparisons and alternatives

After the native worker exits its normal while-not-_stop loop, _generate calls model_provider.reset(), deletes self.prompt_cache, and runs gc.collect() in the generation thread under its stream. These statements are after the loop rather than a universal finally around _generate, so they do not establish cleanup after every exception or immediate release to the operating system. (low confidence) (verified as of 2026-10-23) [source]
Native cancellation is cooperative at explicit checks rather than a demonstrated in-flight Metal interrupt. Batched _generate shares removal UIDs across distributed workers before remove(); sequential _serve_single instead raises NotImplementedError if it observes ctx._should_stop while _is_distributed is true. That exception is caught and sent to the response queue. Distributed and sequential stop handling therefore have different limits. (low confidence) (verified as of 2026-10-23) [source]
For image requests, mlx_lm.server builds a modified cache key by replacing each run of image patch-token IDs with a negative ID derived from SHA-256 of the URI string passed to _make_vision. The HTTP handler first rewrites remote URLs as data URLs, so the hash input for remote images is the normalized data URI rather than the remote host path. The server performs image decoding and processor work before looking up the nearest cached prefix; after a hit, it calls model.embed_inputs only for images whose patch starts fall after the cached prefix. Thus this path can reuse KV and visual embedding work for cached image blocks, while still repeating earlier image input preparation on later requests. (low confidence) (verified as of 2026-10-23) [source]
ResponseGenerator._generate explicitly makes a request nonbatchable when _image_parts(request.messages) finds image content, even if other batching conditions hold. It drains any incompatible active batch, then routes that request through _serve_single; the pinned native image path is sequential rather than an image-capable BatchGenerator path. (low confidence) (verified as of 2026-10-23) [source]

Facts and statements

In the current upstream snapshot, mlx_lm.generate.stream_generate returns a lazy Python generator and its signature has no request ID or explicit cancellation-token argument; callers must provide cancellation through the code that owns and controls iteration. (high confidence) (verified as of 2026-10-23) [source]
LM Studio's adapter adds request-scoped cancellation around MLX-LM: sequential generation uses a request ID mapped to a threading.Event, checks it while waiting for the generation lock and between iterator steps, closes the underlying stream, and removes the pending-request entry; batched generation removes the matching BatchGenerator sequence and reports RequestCancelled. Adapter shutdown joins its worker, drops the model reference, runs garbage collection, and clears the MLX allocator cache. (high confidence) (verified as of 2026-10-23) [source]
A thread-backed server needs to propagate disconnect cancellation to its generation worker: the mlx-openai-server issue reports that Starlette stopped consuming the async iterator while the worker kept decoding, and its proposed fix sets a threading.Event from the async generator's finally block for the worker to check. LM Studio's client documentation also treats cancel() as an explicit operation and warns that simply breaking iteration loses the partial result and final prediction statistics. (medium confidence) (verified as of 2026-10-23) [source]
In MLX-LM source at 0d4d3d04f8ab checked on 2026-10-23, stream_generate is a Python generator with no request ID or explicit cancellation-token parameter; generate_step accepts an optional prompt_cache and updates it in place. The stock server distinguishes ResponseGenerator._stop, which stops its generation thread, from per-request GenerationContext._should_stop and stop(). Its generation paths check the request flag, and the HTTP response handler calls ctx.stop() in finally. These mechanisms do not establish a measured disconnect-to-stop latency or immediate interruption of an MLX evaluation already in progress. (medium confidence) (verified as of 2026-10-23) [source]
LM Studio's mlx-engine implements request-scoped cancellation in both generation modes: its sequential iterator close() calls cancel_request(request_id) and closes the underlying stream in a finally block; its batched remove(request_id) queues cancellation, removes the batch UID, and sends RequestCancelled to that request's response queue. (high confidence) (verified as of 2026-10-23) [source]
LM Studio's TypeScript SDK exposes caller cancellation through OngoingPrediction.cancel() or an AbortController signal and reports userStopped; this is a serving/client API around the model runtime, not a cancellation parameter on MLX-LM's core generator. (medium confidence) (verified as of 2026-10-23) [source]
A cubist38 server report for v1.7.0 found that a client disconnect left its background worker generating until completion unless the wrapper propagated cancellation. The proposed fix set a threading.Event in the async stream's finally block and checked it in the worker loop; the report says this stopped generation within one token. (medium confidence) (verified as of 2026-03-31) [source]
A later cubist38 report for mlx-openai-server v1.8.1 and mlx_lm 0.31.1 said the continuous-batching path could ignore client disconnects, run to natural EOS, or leave a batch slot hung despite cancellation bookkeeping. Treat disconnect cancellation as path-specific and verify the real batched request path. (medium confidence) (verified as of 2026-05-13) [source]
As checked on 2026-10-23, upstream ModelProvider._load() drops model references, runs gc.collect(), and calls mx.clear_cache() before loading a replacement. The server keeps LRUPromptCache separately and the model-load path does not trim that cache; issue #1757 reported that retained prompt-cache KV entries can keep old-model memory alive, so allocator clearing and prompt-cache eviction are separate lifetimes. (medium confidence) (verified as of 2026-10-23) [source]
A Sep 14, 2026 report from a paged-KV benchmark describes KVCache.__del__ calling close() on a garbage-collection thread that does not own the MLX stream, raising RuntimeError: There is no Stream(gpu, 3) in current thread. The report is specific to a paged-cache implementation and does not establish that the standard upstream cache has the same teardown bug. (low confidence) (verified as of 2026-09-14) [source]
A Sep 5, 2026 paged-KV proposal describes page reference counting and cleanup on completion, cancellation, and errors, with local validation; it asks for upstream direction before preparing a PR. Treat it as prototype evidence, not a shipped MLX-LM request-cancellation API. (medium confidence) (verified as of 2026-09-05) [source]
ResponseGenerator.generate queues (response_queue, request, generation_args) and waits for the worker to return a GenerationContext. In this pinned _generate/_next_request path there is no queued-request _should_stop check before context creation. The caller receives the native stop handle only after the worker creates and queues ctx; do not assume pre-dequeue cancellation is implemented. (low confidence) (verified as of 2026-10-23) [source]
In batched _generate, cancellation is observed after batch_generator.next() returns prompt progress or generated responses: the worker reads result["ctx"]._should_stop and collects the UID. In sequential _serve_single, the prefill progress callback only queues counts; it does not inspect ctx._should_stop. That sequential flag is checked after a generated response is queued and its token appended to cache_key, then non-distributed execution breaks the generation loop. (low confidence) (verified as of 2026-10-23) [source]
For stopped native batch requests, _generate shares uids_to_remove, calls batch_generator.remove(uids_to_remove), and pops their batch_results entries. BatchGenerator.remove defaults return_prompt_caches=False and removes matching queued, prompt, or generation entries. The different text-stop finish path explicitly requests return_prompt_caches=True and inserts those returned caches into LRUPromptCache; stop-flag removal does not take that path, while previously inserted segment caches are separate entries. (medium confidence) (verified as of 2026-10-23) [source]
The standalone mlx_lm.cache_prompt CLI builds a cache from one --prompt string or stdin: it applies the tokenizer chat template or encodes the string, runs generate_step with a KV cache, and saves that state. It does not call an image processor or pass pixel_values, so this disk-cache CLI should not be assumed to store image prompt processing state. (medium confidence) (verified as of 2026-10-23) [source]
Image reuse across VLM integrations needs more than token-prefix KV matching. Current LM Studio batched vision code restores prompt caches using prepared image_spans and rebuilds cached prompt kwargs. A mlx-vlm issue describing Qwen3.5 support says image checkpoint reuse required image_end_index and get_partial_input_embeddings so the engine could resume after an image block and encode only new images. A Rapid-MLX report says its engine reran the vision tower until it forwarded mlx-vlm's vision_cache and _image_key contract, which the report says applies only to model families that implement it. Taken together, these sources indicate that a text or KV-prefix hit alone does not prove that every engine/model path skipped vision encoding. (medium confidence) (verified as of 2026-10-23) [source]
Before prompt tokenization, `mlx_lm.server` fetches HTTP(S) image URLs and rewrites them as base64 data URLs; it decodes image bytes to RGB arrays and passes those images with rendered chat text through the model processor. The OpenAI API reference documents image URLs and base64 image data as valid `image_url.url` values. (high confidence) (verified as of 2026-10-23) [source]
Image-cache behavior is serving-backend-specific: LM Studio's batched vision backend initializes a `VlmPromptCacheStore`, a prompt-cache coordinator, and a `VisionFeatureMemoizer`. An April 2026 issue reported that an earlier VisionAddOn/Qwen3.5 path re-prefilled every turn and that `LRUPromptCache` missed some hybrid Qwen3.6 system-plus-user prefixes. (medium confidence) [source]
Muse Spark Processor.__call__ splits rendered chat text on <|patch|> and requires exactly one placeholder per supplied image; a mismatch raises ValueError with the placeholder and image counts. preprocess_image resizes/normalizes each RGB image and returns channels-first pixels plus its token-grid size. Each placeholder becomes <|image_start|>, that many <|patch|> tokens, and <|image_end|>. (low confidence) (verified as of 2026-10-23) [source]
In _serve_single, image-aware prefix reuse counts vision.starts positions before ctx.prompt_cache_count and slices vision.pixel_values[n:] accordingly. Remaining pixels go through model.embed_inputs(mx.array(rest), pixel_values=...), the embeddings are evaluated, and stream_generate receives them as input_embeddings alongside prompt_cache=cache and the uncached prompt suffix. stream_generate forwards its remaining options to generate_step for the ordinary generation path. (medium confidence) (verified as of 2026-10-23) [source]
In MTP speculative decoding, a stateful processor can mutate on a candidate token that is later rejected; if its state is not restored or rebuilt from the accepted prefix, grammar or counter state can drift from the generated history. (high confidence) [source]
MLX-LM's MTP work corrected stale prev_tokens handling and rejected-draft history separately from processor state. The thinking-budget processor can still advance during verification for a bonus token that is later rejected; the maintainer says recomputing state from accepted token history or adding an explicit save_state()/restore_state() protocol would prevent counter drift. (high confidence) (verified as of 2026-10-23) [source]
The current mlx-lm main speculative_generate_step calls logits processors during draft and verification _step calls, then rejection cleanup truncates prev_tokens and trims model caches; the public LogitsProcessor alias is only Callable[[mx.array, mx.array], mx.array], so the interface exposes no generic processor-state rollback hook. (high confidence) (verified as of 2026-10-23) [source]
Two remedies are described for stateful processors: recompute state from the accepted token history on each call, or add processor state snapshot and restore around speculative verification; the latter is a proposed interface need, not a generic method in the reviewed MLX-LM callable API. (medium confidence) (verified as of 2026-10-23) [source]
Independent MTP guidance also treats this as accepted-prefix bookkeeping: Metrale says sampler state must track accepted tokens and rejected drafts must be unwound, while an OMLX MTP discussion says a grammar matcher must advance on accepted tokens and roll back on rejected drafts. (high confidence) [source]
In speculative_generate_step at 0d4d3d04f8ab, _draft_generate calls _step(draft_model, draft_cache, y) for each proposal. _step appends y to prev_tokens and invokes _process_and_sample(prev_tokens, logits), which applies processors in list order before logsumexp normalization and sampling. Thus processors run while proposals are being formed, not only for accepted output tokens. (low confidence) (verified as of 2026-10-23) [source]
Before main-model verification, speculative_generate_step truncates prev_tokens to remove the draft-side context extension and calls _step(model, model_cache, concatenated_input_and_drafts, num_draft+1). For n_predict>1, _step first shortens y by n_predict-1 positions; on each prediction it appends the current y to prev_tokens, calls processors with that history and logits[:, i, :], and uses the sampled main-model y for the next prediction. These are the pinned implementation inputs, not a guarantee that every processor call corresponds to an accepted token. (low confidence) (verified as of 2026-10-23) [source]

Children

MLX-LM server worker supervision after generation failures (frontier)
Request cancellation in MLX serving wrappers (frontier)
Image-part normalization and model processor inputs (frontier)
URI-derived image identity in prefix cache keys (frontier)
Visual feature caches versus language-model KV caches (frontier)
Stateful logits processor save and restore contracts (frontier)
MTP accepted-prefix semantics for grammar and sampler state (frontier)
MLX-LM MTP accepted-token history (frontier)
Logits processor state snapshot and restore contract (frontier)
MLX-LM native request cancellation and generation resource lifecycle
MLX-LM native vision inputs and image-aware prompt caching
MLX-LM stateful logits processor rollback during speculative decoding

← the whole tree · 3D view· how to read this page