MLX-LM native request cancellation and generation resource lifecycle
Parent: MLX-LM native multimodal generation and request lifecycle · Published reference · snapshot 2026-10-23 · viability 4.30 / 5 (research)
↓ Facts as markdownall context files
Source-anchored research for MLX-LM native request cancellation and generation resource lifecycle. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
How it works
- The current mlx_lm.server keeps an LRUPromptCache on its ResponseGenerator and inserts extracted prompt-segment KV caches into it. An issue report describes BrokenPipeError on disconnected responses while request sequences remained in the cache, but the reporter says they could not establish whether disconnections caused or accelerated the deadlock; a disconnect should not be assumed to free cached KV or be treated as a proven root cause. (medium confidence) (verified as of 2026-10-23) [source]
- At pinned 0d4d3d04f8ab, APIHandler.handle_completion observes connection failures when wfile.write or flush raises during keepalive, SSE, or final response output; it does not poll for disconnects. ResponseGenerator.generate first returns ctx and a lazy _inner response iterator. Only consumption of that iterator invokes keepalive_callback on queued progress tuples. handle_completion consumes it inside the response try block, whose finally calls ctx.stop(), including when a keepalive or response write fails there. (low confidence) (verified as of 2026-10-23) [source]
Problems, failure modes and limitations
- stream_generate enters wired_limit around generation. When maybe_set_recommended_wired_limit returns a prior limit, wired_limit synchronizes the supplied streams (or the default synchronization path) and restores that limit in finally. If the helper returns None, wired_limit yields and returns without that synchronization or restoration. Closing or exhausting an entered generator unwinds its context; LM Studio's sequential adapter explicitly closes its wrapped stream in finally. (high confidence) (verified as of 2026-10-23) [source]
Comparisons and alternatives
- After the native worker exits its normal while-not-_stop loop, _generate calls model_provider.reset(), deletes self.prompt_cache, and runs gc.collect() in the generation thread under its stream. These statements are after the loop rather than a universal finally around _generate, so they do not establish cleanup after every exception or immediate release to the operating system. (low confidence) (verified as of 2026-10-23) [source]
- Native cancellation is cooperative at explicit checks rather than a demonstrated in-flight Metal interrupt. Batched _generate shares removal UIDs across distributed workers before remove(); sequential _serve_single instead raises NotImplementedError if it observes ctx._should_stop while _is_distributed is true. That exception is caught and sent to the response queue. Distributed and sequential stop handling therefore have different limits. (low confidence) (verified as of 2026-10-23) [source]
Facts and statements
- In the current upstream snapshot, mlx_lm.generate.stream_generate returns a lazy Python generator and its signature has no request ID or explicit cancellation-token argument; callers must provide cancellation through the code that owns and controls iteration. (high confidence) (verified as of 2026-10-23) [source]
- LM Studio's adapter adds request-scoped cancellation around MLX-LM: sequential generation uses a request ID mapped to a threading.Event, checks it while waiting for the generation lock and between iterator steps, closes the underlying stream, and removes the pending-request entry; batched generation removes the matching BatchGenerator sequence and reports RequestCancelled. Adapter shutdown joins its worker, drops the model reference, runs garbage collection, and clears the MLX allocator cache. (high confidence) (verified as of 2026-10-23) [source]
- A thread-backed server needs to propagate disconnect cancellation to its generation worker: the mlx-openai-server issue reports that Starlette stopped consuming the async iterator while the worker kept decoding, and its proposed fix sets a threading.Event from the async generator's finally block for the worker to check. LM Studio's client documentation also treats cancel() as an explicit operation and warns that simply breaking iteration loses the partial result and final prediction statistics. (medium confidence) (verified as of 2026-10-23) [source]
- In MLX-LM source at 0d4d3d04f8ab checked on 2026-10-23, stream_generate is a Python generator with no request ID or explicit cancellation-token parameter; generate_step accepts an optional prompt_cache and updates it in place. The stock server distinguishes ResponseGenerator._stop, which stops its generation thread, from per-request GenerationContext._should_stop and stop(). Its generation paths check the request flag, and the HTTP response handler calls ctx.stop() in finally. These mechanisms do not establish a measured disconnect-to-stop latency or immediate interruption of an MLX evaluation already in progress. (medium confidence) (verified as of 2026-10-23) [source]
- LM Studio's mlx-engine implements request-scoped cancellation in both generation modes: its sequential iterator close() calls cancel_request(request_id) and closes the underlying stream in a finally block; its batched remove(request_id) queues cancellation, removes the batch UID, and sends RequestCancelled to that request's response queue. (high confidence) (verified as of 2026-10-23) [source]
- LM Studio's TypeScript SDK exposes caller cancellation through OngoingPrediction.cancel() or an AbortController signal and reports userStopped; this is a serving/client API around the model runtime, not a cancellation parameter on MLX-LM's core generator. (medium confidence) (verified as of 2026-10-23) [source]
- A cubist38 server report for v1.7.0 found that a client disconnect left its background worker generating until completion unless the wrapper propagated cancellation. The proposed fix set a threading.Event in the async stream's finally block and checked it in the worker loop; the report says this stopped generation within one token. (medium confidence) (verified as of 2026-03-31) [source]
- A later cubist38 report for mlx-openai-server v1.8.1 and mlx_lm 0.31.1 said the continuous-batching path could ignore client disconnects, run to natural EOS, or leave a batch slot hung despite cancellation bookkeeping. Treat disconnect cancellation as path-specific and verify the real batched request path. (medium confidence) (verified as of 2026-05-13) [source]
- As checked on 2026-10-23, upstream ModelProvider._load() drops model references, runs gc.collect(), and calls mx.clear_cache() before loading a replacement. The server keeps LRUPromptCache separately and the model-load path does not trim that cache; issue #1757 reported that retained prompt-cache KV entries can keep old-model memory alive, so allocator clearing and prompt-cache eviction are separate lifetimes. (medium confidence) (verified as of 2026-10-23) [source]
- A Sep 14, 2026 report from a paged-KV benchmark describes KVCache.__del__ calling close() on a garbage-collection thread that does not own the MLX stream, raising RuntimeError: There is no Stream(gpu, 3) in current thread. The report is specific to a paged-cache implementation and does not establish that the standard upstream cache has the same teardown bug. (low confidence) (verified as of 2026-09-14) [source]
- A Sep 5, 2026 paged-KV proposal describes page reference counting and cleanup on completion, cancellation, and errors, with local validation; it asks for upstream direction before preparing a PR. Treat it as prototype evidence, not a shipped MLX-LM request-cancellation API. (medium confidence) (verified as of 2026-09-05) [source]
- ResponseGenerator.generate queues (response_queue, request, generation_args) and waits for the worker to return a GenerationContext. In this pinned _generate/_next_request path there is no queued-request _should_stop check before context creation. The caller receives the native stop handle only after the worker creates and queues ctx; do not assume pre-dequeue cancellation is implemented. (low confidence) (verified as of 2026-10-23) [source]
- In batched _generate, cancellation is observed after batch_generator.next() returns prompt progress or generated responses: the worker reads result["ctx"]._should_stop and collects the UID. In sequential _serve_single, the prefill progress callback only queues counts; it does not inspect ctx._should_stop. That sequential flag is checked after a generated response is queued and its token appended to cache_key, then non-distributed execution breaks the generation loop. (low confidence) (verified as of 2026-10-23) [source]
- For stopped native batch requests, _generate shares uids_to_remove, calls batch_generator.remove(uids_to_remove), and pops their batch_results entries. BatchGenerator.remove defaults return_prompt_caches=False and removes matching queued, prompt, or generation entries. The different text-stop finish path explicitly requests return_prompt_caches=True and inserts those returned caches into LRUPromptCache; stop-flag removal does not take that path, while previously inserted segment caches are separate entries. (medium confidence) (verified as of 2026-10-23) [source]
Children
- MLX-LM server worker supervision after generation failures (frontier)
- Request cancellation in MLX serving wrappers (frontier)
Frontier under this node: MLX-LM server worker supervision after generation failures, Request cancellation in MLX serving wrappers