MLX-LM native service model lifecycle and release contracts

Parent: MLX and mlx-lm on Apple silicon · Published reference · snapshot 2026-10-23

Source-anchored research for MLX-LM native service model lifecycle and release contracts. Run status: COMPLETED.

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Definitions

If stream_options is an object without include_usage, such as {}, the handler indexes self.stream_options["include_usage"] after it has sent the finish chunk. That raises KeyError before data: [DONE]. OpenAI documents include_usage as optional, so an empty stream_options object is valid under that schema but does not complete normally in this code path. (high confidence) (verified as of 2026-10-23) [source]

Structure and components

The current manager defaults its pattern to mlx and both scan and delete select entries when that substring occurs anywhere in repo.repo_id. A plain scan therefore omits cached repos whose IDs do not contain mlx. (high confidence) (verified as of 2026-10-23) [source]
In pinned pyproject.toml, [project].requires-python sets the Python floor to >=3.11. [project].dependencies contains "mlx>=0.32.3; platform_system == 'Darwin'" and "transformers>=5.7.0". Under [project.optional-dependencies], train supplies datasets and tqdm; evaluate supplies lm-eval and tqdm. These are the pinned main requirements, not a replacement for the v0.32.2 wheel metadata. (low confidence) (verified as of 2026-10-23) [source]

How it works

A model-type alias is valid only when the model graph and weight semantics match. The oMLX Xing4.0 report describes a distinct Xing4_0HyperConnection residual topology and says aliasing xing4_0 to deepseek_v3 would produce incorrect outputs; it calls for a compatible implementation and sanitizer instead. (medium confidence) [source]
MLX-LM imports a config-declared model_file from the model directory only when trust_remote_code=True; the default is false. The merged upstream change threads this opt-in through load() and sharded_load(), and oMLX preflights the same trust decision before starting MLX-LM loading. (high confidence) (verified as of 2026-10-23) [source]
A Hugging Face model repo ID passed to an MLX-LM generation command is downloaded from the Hub when it is not already cached. Conversion writes the resulting MLX model to mlx_model by default, separate from the Hub source cache. (high confidence) (verified as of 2026-10-17) [source]
Hugging Face revision deletion removes the selected snapshot and its refs, deletes a blob only when no remaining cached revision references it, and removes the repo cache when no revisions remain. (high confidence) (verified as of 2026-10-23) [source]
For caches using Xet shared blobs, Hugging Face warns that older cache clients can leave shared payloads behind or delete a blob still referenced by another repository. Because mlx_lm.manage delegates deletion to the installed Hugging Face delete_revisions() helper, physical cleanup behavior depends on that runtime helper; the sources do not establish one result for every installation. (medium confidence) (verified as of 2026-10-23) [source]
Issue #1493 does not establish a generic prompt-length or stream-only trigger: synthetic requests around the same long context, including a streaming request, passed. The later report captured a no-delivery livelock in the Gemma case; the uncaught OOM worker-exit explanation concerned a separate Qwen server. Five replay turns passed, but the reporter explicitly said that replay did not validate watchdog recovery. (low confidence) [source]
When stream_options.include_usage is enabled, normal MLX-LM stream chunks omit the usage field entirely: generate_response adds usage only to non-stream responses, and the handler emits usage separately in a terminal chunk. OpenAI's documented Chat Completions contract says the other chunks include usage: null when the final usage chunk is requested, so clients that depend on this field see a compatibility difference. (high confidence) (verified as of 2026-10-23) [source]
Inference: reproducible deployments should lock mlx-lm, mlx, transformers, and Python together, record the resolved versions, and rerun workload tests before point upgrades because the published and main-branch MLX floors differ and a downstream project excluded one point release after A/B testing. (medium confidence) [source]
A reproducible result needs both the MLX-LM software version or source commit and the model revision. The former identifies runtime/training implementation; load(..., revision=<branch,tag,commit>) selects model files from the Hub. Use an immutable model commit when recording exact weights; a package-version pin does not pin those model files. (medium confidence) (verified as of 2026-10-23) [source]

How-to and procedures

To install the comparison release explicitly, use python -m pip install "mlx-lm==0.32.2" in the intended environment. The == specifier fixes the mlx-lm package version; it does not by itself freeze every transitive dependency. Record the resolved environment alongside the package pin. (medium confidence) (verified as of 2026-10-23) [source]

Measurements and reference values

The captured vLLM Metal manifest pins an MLX-LM Git commit that its dependency comment dates to 2026-08-25 and says provides cache APIs absent from 0.31.3. That comment does not establish when vLLM Metal adopted the pin. The October 4 mlx-dspark changelog reports that MLX-LM 0.32.0 removed meta_state and records a cache-snapshot fix supporting both 0.31.3 and 0.32.0. (medium confidence) (verified as of 2026-10-04) [source]

Problems, failure modes and limitations

load_model reads model*.safetensors, applies optional model_config overrides, builds ModelArgs.from_dict(config) and Model, calls model.sanitize(weights) when provided, then calls model.load_weights(..., strict=True) by default. A registry mapping alone therefore does not adapt checkpoint tensor names or layouts that the selected model cannot consume. (high confidence) (verified as of 2026-10-23) [source]
Current native error handling depends on when failure occurs. Request validation errors return JSON bodies with HTTP 400 or 411; an exception from response_generator.generate() before headers are sent returns an error JSON body with HTTP 404. After the server sends HTTP 200 and text/event-stream, handle_completion has a finally cleanup but no explicit SSE error-frame or [DONE] path for an exception, so a mid-stream failure can appear truncated. A historical mlx-lm 0.31.3 report also describes generation failures that surfaced no client error. (high confidence) (verified as of 2026-10-23) [source]
Current main polls each response queue with a one-second timeout and raises RuntimeError saying the generation thread died when the generation thread is unavailable, so it no longer has the dead-thread indefinite-wait behavior described for mlx-lm 0.31.3 in issue #1493. Its health check tests thread liveness and a failure flag; it does not detect a live worker that stops delivering responses. (medium confidence) (verified as of 2026-10-23) [source]
A separate mlx-openai-server issue shows a generation failure returned inside an SSE data: error object with the message 'Failed to generate text stream: There is no Stream(gpu, 1) in current thread.' That is a downstream wrapper's error envelope; current native mlx_lm.server instead writes JSON errors before streaming starts and has no explicit post-header SSE error encoder. (medium confidence) (verified as of 2026-10-23) [source]
Before stream headers are sent, missing Content-Length returns 411; a nonnumeric length, invalid JSON, non-object JSON body, parameter validation error, or request-factory exception returns 400. An unknown POST route returns 404. An exception from generation setup also returns 404 with JSON shaped as {"error": "<message>"}, even for a streaming request; OpenAI's error guide describes NotFoundError as a missing resource, so status-based clients can misclassify a model or setup failure as a not-found error. (medium confidence) (verified as of 2026-10-23) [source]
Native server POST routes are /v1/completions for text and /v1/chat/completions or /chat/completions for chat. Chat bodies require messages, while text bodies require prompt; their request factories assert those keys and the POST handler converts request-factory exceptions into a 400 JSON error. (low confidence) (verified as of 2026-10-23) [source]
Set stream=true for an SSE response. The server terminates with data: [DONE]; stream_options.include_usage adds a separate usage packet before that sentinel. Missing Content-Length returns 411; invalid JSON returns 400 with an "Invalid JSON in request body" error. (low confidence) (verified as of 2026-10-23) [source]
The deprecated direct python -m mlx_lm.lora form prints a replacement notice: use mlx_lm.lora ... or python -m mlx_lm lora .... The package manifest supplies the mlx_lm.lora console script. (medium confidence) (verified as of 2026-10-23) [source]

Comparisons and alternatives

load_model accepts a get_model_classes callable that maps config to (Model, ModelArgs) and defaults to _get_classes. The higher-level load() signature does not expose or forward this callback, so it is a resolver hook for direct load_model callers rather than a global loader registry. (low confidence) (verified as of 2026-10-23) [source]
`mlx_lm.manage` enumerates Hub repository revisions and delegates deletion to `delete_revisions()`, rather than exposing dedicated partial-download maintenance. Current Hugging Face documentation assigns partial-download and detached-revision cleanup to `hf cache prune`; it also says `hf cache ls` flags incomplete downloads and `hf cache rm` can remove partial files when deleting an entire repository. Revision-oriented management therefore does not imply that every repository deletion leaves .incomplete files behind. (high confidence) (verified as of 2026-10-23) [source]
If the configured Hub cache directory does not exist, scan_cache_dir() raises CacheNotFound. Since mlx_lm.manage --scan calls it directly, a fresh install with no cache directory can fail instead of returning an empty listing; MLX-LM issue 632 reports this behavior and its maintainer says the exception is expected. (high confidence) (verified as of 2026-10-23) [source]
For --delete, a blank response means no; only y, yes, or 1 confirms. After confirmation, MLX-LM passes every revision hash for each matched repository to HFCacheInfo.delete_revisions(), so the command removes matching repo revisions rather than selecting one file. (high confidence) (verified as of 2026-10-23) [source]
Current main marks the background generation worker unavailable after an uncaught worker failure, and GET /health then returns 503; GET /v1/models remains a metadata route that returns 200. A 2026-08 report on mlx-lm 0.31.3 observed /v1/models returning 200 while a generation thread had died and streams hung, so that report describes a historical failure mode rather than the current /health response contract. (medium confidence) (verified as of 2026-10-23) [source]
The HTTP sampling validator accepts numeric top_p in the inclusive interval [0,1]: values below zero or above one raise ValueError. The request field model chooses the model (default "default_model"), and the plural field adapters chooses an optional adapter path (default None). These names differ from the --adapter-path CLI option. (low confidence) (verified as of 2026-10-23) [source]
On Python 3.10, pip can select mlx-lm 0.31.3 rather than fail because 0.32.2 requires Python >=3.11; a third-party install check observed 0.31.3 as the last release accepted by Python 3.10, so verify the interpreter and installed package version after installation. (medium confidence) (verified as of 2026-10-17) [source]

Facts and statements

In MLX-LM's built-in path, _get_classes requires config['model_type'], optionally remaps a (model_type, architectures) pair or a model-type alias, imports mlx_lm.models.<resolved-type>, and expects that module to expose Model and ModelArgs. A repository name or architecture label alone does not add an implementation to the loader. (high confidence) (verified as of 2026-10-23) [source]
load_model has a separate config model_file path: it imports that file and expects Model and ModelArgs. This route raises unless trust_remote_code=True, which defaults to false. A Qwen3.8 checkpoint repository demonstrates stock mlx_lm.load(..., trust_remote_code=True) using its bundled model_file for an architecture absent from the released registry. (high confidence) (verified as of 2026-10-23) [source]
Model-port validation needs reference outputs as well as model-type and shape checks: porting guidance describes tensor transposes and rotary/cache conventions that preserve shapes but change outputs, while an independent Qwen3.8 port reports sanitization, normalization, hash-seed, and sparse-mask errors in a candidate upstream implementation. (high confidence) (verified as of 2026-10-23) [source]
The minimum architecture metadata is consumer-specific: MLX-LM requires model_type and treats architectures as optional remapping input, while Ollama issue #19128 reports that its MLX importer and runner rejected a checkpoint with model_type but no architectures[0], even though the model_type named a supported runner architecture. Services should define and validate metadata fallbacks for each backend. (medium confidence) (verified as of 2026-10-19) [source]
_get_classes reads config.model_type and examines architectures in order, treating a string as one architecture. The first matching (model_type, architecture) entry in MODEL_ARCHITECTURE_REMAPPING wins; if none matches, MODEL_REMAPPING applies. It imports mlx_lm.models.<resolved_type> and returns that module's Model and ModelArgs. ImportError becomes ValueError("Model type <resolved_type> not supported."). (low confidence) (verified as of 2026-10-23) [source]
load_model reads config.json, then calls config.update(model_config) for supplied overrides before ModelArgs.from_dict and Model construction. This is a shallow dictionary update: supplied values replace values under matching top-level keys. A model sanitize hook transforms the loaded weight dictionary before the final model.load_weights call. (low confidence) (verified as of 2026-10-23) [source]
With strict=True, load_model raises FileNotFoundError("No safetensors found in <model_path>") if model*.safetensors matches no files. It passes strict to model.load_weights; the documented contract raises if weights do not match the model. strict=False relaxes that contract and the missing-weight-files guard. (low confidence) (verified as of 2026-10-23) [source]
For config.quantization, the loader first honors explicit per-module entries keyed by module path. Otherwise class_predicate requires a to_quantized method and a matching <path>.scales checkpoint tensor, then infers the module quantization settings from checkpoint shapes/dtypes. A supported global_scale tensor adds that argument. This selective conversion prepares modules before weights are loaded. (low confidence) (verified as of 2026-10-23) [source]
mlx_lm.manage calls huggingface_hub.scan_cache_dir() without a custom cache path. The Hub cache defaults to ~/.cache/huggingface/hub; HF_HUB_CACHE can set the cache directory directly and HF_HOME changes its parent. Hub clients using the same configuration can share those cached repo files. (high confidence) (verified as of 2026-10-23) [source]
The Hub cache stores each model repo under a models--<org>--<repo> directory. It stores commit revisions in snapshots and branch or tag pointers in refs; snapshot files point to blobs, so unchanged files can be shared across revisions without duplicate bytes. (high confidence) (verified as of 2026-10-23) [source]
mlx_lm.manage --delete applies the same pattern to repo IDs, displays the matching repos with their sizes and local paths, then asks for yes/no confirmation. On confirmation it processes every cached revision it found for each matched repo; a no or blank answer cancels deletion. (high confidence) (verified as of 2026-10-23) [source]
The current implementation filters only on repo.repo_id and does not restrict repo_type. A dataset or Space whose repo ID matches the pattern can therefore appear in --scan and be selected by --delete, even though the CLI help describes models. (high confidence) (verified as of 2026-10-23) [source]
The default Hub cache is ~/.cache/huggingface/hub, and HF_HOME or HF_HUB_CACHE can customize its location. Hugging Face describes this as a central cache shared across Hub libraries; LocalAIScope likewise notes that other Hugging Face tools on a Mac share it. Removing a matching entry through MLX therefore changes the cache used by those tools too. (high confidence) [source]
Hub snapshots represent repository revisions and point to content-addressed blobs, which may be shared among revisions or repositories. Hugging Face distinguishes logical per-repository sizes from physical cache size, so the per-repository SIZE ON DISK shown by mlx_lm.manage --scan is not guaranteed to equal the physical space reclaimed by deletion. (high confidence) (verified as of 2026-10-23) [source]
mlx_lm.manage --scan displays SIZE ON DISK from repo.size_on_disk_str and LOCAL PATH from repo.repo_path. --pattern performs a substring test against repo.repo_id. With --delete, ask_for_confirmation treats an empty response as no and prints "Deletion cancelled - no changes made."; if no repository matches, it prints "No models found matching pattern" with the pattern. (low confidence) (verified as of 2026-10-23) [source]
load(..., revision=...) forwards revision to the Hub download helper for a nonlocal model. The supported revision identifier can be a branch name, tag, or commit hash. An existing local model path bypasses the Hub download, so a revision string does not select a different checkpoint within that local directory. (low confidence) (verified as of 2026-10-23) [source]
For POST /v1/chat/completions with stream=true, the current handler returns HTTP 200 with Content-Type: text/event-stream and Cache-Control: no-cache. It flushes each JSON chunk as a data: record ending in a blank line, then emits a final chunk carrying finish_reason, an optional usage-only chunk, and data: [DONE]. This matches WHATWG event framing and the documented Chat Completions finish and usage chunk ordering. (high confidence) (verified as of 2026-10-23) [source]
A vllm-mlx 0.4.0 issue reported that natural EOS streaming ended with finish_reason:null followed by [DONE], while token-cap stops emitted length; a strict OpenAI-compatible client rejected the natural-stop result. The issue was closed as fixed on 2026-08-12. This is an adjacent MLX-based implementation, not evidence that native mlx_lm.server had the same defect. (medium confidence) (verified as of 2026-08-12) [source]
A downstream mlx-openai-server v1.7.0 report found that disconnecting an SSE client stopped stream consumption but did not signal its inference worker; generation continued until completion and blocked the request queue. The proposed threading.Event cancellation stopped generation within one token. This describes that wrapper's behavior, not native mlx_lm.server. (low confidence) [source]
The handler has no exception-to-SSE branch around response iteration. If generation raises after streaming has started, control skips the final finish chunk, optional usage chunk, and [DONE], although the context is stopped in finally. Clients should treat stream termination without [DONE] as incomplete; OpenAI notes that interruption may omit the final usage chunk, and WHATWG discards an event that reaches EOF without its terminating blank line. (high confidence) (verified as of 2026-10-23) [source]
On 2026-10-16, mlx-lm 0.32.2 was the latest PyPI release; 0.32.0 shipped October 1, 0.32.1 October 15, and 0.32.2 October 16, after 0.31.3 on April 22. (high confidence) (verified as of 2026-10-16) [source]
The published mlx-lm 0.32.2 metadata requires Python >=3.11, mlx>=0.32.2 on Darwin, and transformers>=5.7.0; its cpu, cuda12, and cuda13 extras also require the corresponding mlx backend at >=0.32.2. (high confidence) (verified as of 2026-10-16) [source]
The 0.32.2 release source sets the Darwin MLX floor to >=0.32.2, while upstream main has raised that floor to >=0.32.3 and raised the matching cpu, cuda12, and cuda13 extra floors too. The moving main manifest therefore does not describe the dependency contract of the published 0.32.2 release. (high confidence) (verified as of 2026-10-23) [source]
Chad's pyproject, checked 2026-10-23, keeps Darwin mlx-lm in the 0.31 series (>=0.31.3,<0.32) while bounding MLX independently to >=0.32.0, !=0.32.1, and <=0.32.2. Its notes say MLX 0.32.1 caused degenerate generations with its custom kernels, confirmed by an A/B test on 2026-08-23, and admit 0.32.2 only after its own workload gate; this is application-specific evidence, not an upstream compatibility guarantee. (low confidence) (verified as of 2026-10-23) [source]
On 2026-10-23 the upstream release page marked v0.32.2 as latest; PyPI records its upload on 2026-10-16, after v0.32.0 on October 1 and v0.32.1 on October 15. An independent install guide reported pip resolving mlx-lm 0.32.2 on October 17. (high confidence) (verified as of 2026-10-23) [source]
This comparison uses stable v0.32.2 as its release baseline and commit 0d4d3d04f8ab7ce4a51e7209e8615755a48ba94d as its exact main implementation snapshot. Qualify findings seen only at that main commit as snapshot behavior; they are not established v0.32.2 behavior without checking the release source. The pinned main manifest requires MLX>=0.32.3 on Darwin, while the release manifest requires MLX>=0.32.2. (medium confidence) (verified as of 2026-10-23) [source]

Children

MLX-LM built-in architecture dispatch (frontier)
MLX-LM model_file and trust_remote_code contract (frontier)
MLX-LM architecture-specific weight sanitization and parity validation (frontier)
MLX-LM server SSE terminal chunks and finish_reason semantics (frontier)
MLX-LM native request cancellation on client disconnect (frontier)
Per-release wheel metadata and platform markers (frontier)
Workload-specific point-release gating (frontier)
Published release metadata versus moving branch metadata (frontier)
Workload-specific point-release gates (frontier)
MLX-LM model architecture integration and loader contracts
MLX-LM model management and Hugging Face cache lifecycle
MLX-LM native HTTP request streaming and error contracts
MLX-LM version pinning and current release compatibility delta

← the whole tree · 3D view· how to read this page