MLX-LM native HTTP request streaming and error contracts

Parent: MLX-LM native service model lifecycle and release contracts · Published reference · snapshot 2026-10-23 · viability 4.45 / 5 (research)

↓ Facts as markdown all context files

Source-anchored research for MLX-LM native HTTP request streaming and error contracts. Run status: COMPLETED.

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Definitions

If stream_options is an object without include_usage, such as {}, the handler indexes self.stream_options["include_usage"] after it has sent the finish chunk. That raises KeyError before data: [DONE]. OpenAI documents include_usage as optional, so an empty stream_options object is valid under that schema but does not complete normally in this code path. (high confidence) (verified as of 2026-10-23) [source]

How it works

Issue #1493 does not establish a generic prompt-length or stream-only trigger: synthetic requests around the same long context, including a streaming request, passed. The later report captured a no-delivery livelock in the Gemma case; the uncaught OOM worker-exit explanation concerned a separate Qwen server. Five replay turns passed, but the reporter explicitly said that replay did not validate watchdog recovery. (low confidence) [source]
When stream_options.include_usage is enabled, normal MLX-LM stream chunks omit the usage field entirely: generate_response adds usage only to non-stream responses, and the handler emits usage separately in a terminal chunk. OpenAI's documented Chat Completions contract says the other chunks include usage: null when the final usage chunk is requested, so clients that depend on this field see a compatibility difference. (high confidence) (verified as of 2026-10-23) [source]

Problems, failure modes and limitations

Current native error handling depends on when failure occurs. Request validation errors return JSON bodies with HTTP 400 or 411; an exception from response_generator.generate() before headers are sent returns an error JSON body with HTTP 404. After the server sends HTTP 200 and text/event-stream, handle_completion has a finally cleanup but no explicit SSE error-frame or [DONE] path for an exception, so a mid-stream failure can appear truncated. A historical mlx-lm 0.31.3 report also describes generation failures that surfaced no client error. (high confidence) (verified as of 2026-10-23) [source]
Current main polls each response queue with a one-second timeout and raises RuntimeError saying the generation thread died when the generation thread is unavailable, so it no longer has the dead-thread indefinite-wait behavior described for mlx-lm 0.31.3 in issue #1493. Its health check tests thread liveness and a failure flag; it does not detect a live worker that stops delivering responses. (medium confidence) (verified as of 2026-10-23) [source]
A separate mlx-openai-server issue shows a generation failure returned inside an SSE data: error object with the message 'Failed to generate text stream: There is no Stream(gpu, 1) in current thread.' That is a downstream wrapper's error envelope; current native mlx_lm.server instead writes JSON errors before streaming starts and has no explicit post-header SSE error encoder. (medium confidence) (verified as of 2026-10-23) [source]
Before stream headers are sent, missing Content-Length returns 411; a nonnumeric length, invalid JSON, non-object JSON body, parameter validation error, or request-factory exception returns 400. An unknown POST route returns 404. An exception from generation setup also returns 404 with JSON shaped as {"error": "<message>"}, even for a streaming request; OpenAI's error guide describes NotFoundError as a missing resource, so status-based clients can misclassify a model or setup failure as a not-found error. (medium confidence) (verified as of 2026-10-23) [source]
Native server POST routes are /v1/completions for text and /v1/chat/completions or /chat/completions for chat. Chat bodies require messages, while text bodies require prompt; their request factories assert those keys and the POST handler converts request-factory exceptions into a 400 JSON error. (low confidence) (verified as of 2026-10-23) [source]
Set stream=true for an SSE response. The server terminates with data: [DONE]; stream_options.include_usage adds a separate usage packet before that sentinel. Missing Content-Length returns 411; invalid JSON returns 400 with an "Invalid JSON in request body" error. (low confidence) (verified as of 2026-10-23) [source]

Comparisons and alternatives

Current main marks the background generation worker unavailable after an uncaught worker failure, and GET /health then returns 503; GET /v1/models remains a metadata route that returns 200. A 2026-08 report on mlx-lm 0.31.3 observed /v1/models returning 200 while a generation thread had died and streams hung, so that report describes a historical failure mode rather than the current /health response contract. (medium confidence) (verified as of 2026-10-23) [source]
The HTTP sampling validator accepts numeric top_p in the inclusive interval [0,1]: values below zero or above one raise ValueError. The request field model chooses the model (default "default_model"), and the plural field adapters chooses an optional adapter path (default None). These names differ from the --adapter-path CLI option. (low confidence) (verified as of 2026-10-23) [source]

Facts and statements

For POST /v1/chat/completions with stream=true, the current handler returns HTTP 200 with Content-Type: text/event-stream and Cache-Control: no-cache. It flushes each JSON chunk as a data: record ending in a blank line, then emits a final chunk carrying finish_reason, an optional usage-only chunk, and data: [DONE]. This matches WHATWG event framing and the documented Chat Completions finish and usage chunk ordering. (high confidence) (verified as of 2026-10-23) [source]
A vllm-mlx 0.4.0 issue reported that natural EOS streaming ended with finish_reason:null followed by [DONE], while token-cap stops emitted length; a strict OpenAI-compatible client rejected the natural-stop result. The issue was closed as fixed on 2026-08-12. This is an adjacent MLX-based implementation, not evidence that native mlx_lm.server had the same defect. (medium confidence) (verified as of 2026-08-12) [source]
A downstream mlx-openai-server v1.7.0 report found that disconnecting an SSE client stopped stream consumption but did not signal its inference worker; generation continued until completion and blocked the request queue. The proposed threading.Event cancellation stopped generation within one token. This describes that wrapper's behavior, not native mlx_lm.server. (low confidence) [source]
The handler has no exception-to-SSE branch around response iteration. If generation raises after streaming has started, control skips the final finish chunk, optional usage chunk, and [DONE], although the context is stopped in finally. Clients should treat stream termination without [DONE] as incomplete; OpenAI notes that interruption may omit the final usage chunk, and WHATWG discards an event that reaches EOF without its terminating blank line. (high confidence) (verified as of 2026-10-23) [source]

Children

MLX-LM server SSE terminal chunks and finish_reason semantics (frontier)
MLX-LM native request cancellation on client disconnect (frontier)

Frontier under this node: MLX-LM native request cancellation on client disconnect, MLX-LM server SSE terminal chunks and finish_reason semantics

← the whole tree · 3D view· how to read this page