MLX-LM native vision inputs and image-aware prompt caching

Parent: MLX-LM native multimodal generation and request lifecycle · Published reference · snapshot 2026-10-23 · viability 4.40 / 5 (research)

↓ Facts as markdown all context files

Source-anchored research for MLX-LM native vision inputs and image-aware prompt caching. Run status: COMPLETED.

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Structure and components

In the built-in HTTP server's chat-template path, image content parts marked image_url, input_image, or image are accepted when the loaded model has a processor. The server fetches HTTP(S) images into data URLs, decodes RGB pixels, applies the processor to the templated message, and supplies token IDs plus image embeddings to generation. This includes the common Chat Completions image_url form, which may carry a URL or base64 image data. (high confidence) (verified as of 2026-10-23) [source]
In `mlx_lm.server` chat handling, image content is recognized on the chat-template path when a model processor is loaded; the server preserves parts with type `image_url`, `input_image`, or `image`. OpenAI's Chat Completions reference independently defines user image inputs as content-part arrays with an `image_url.url` field. (high confidence) (verified as of 2026-10-23) [source]
load_processor(model,config) returns None without a truthy vision_config. Otherwise it chooses config.processor_type or the final component of type(model).__module__, looks that name up in PROCESSORS, and invokes processor.from_config(config) when registered. At 0d4d3d04f8ab, PROCESSORS registers only "muse_spark": muse_spark.Processor; this does not establish support for arbitrary multimodal architectures. (low confidence) (verified as of 2026-10-23) [source]
A standard chat image part is {"type":"image_url","image_url":{"url":"<http(s) or data URL>"}} within a messages content list. _image_url extracts its URL; _inline_image_urls fetches remote bytes and replaces them with a base64 data URL before tokenization. _decode_image accepts that data URL, base64-decodes it, opens the bytes with Pillow, converts to RGB, and returns a uint8 NumPy array. (low confidence) (verified as of 2026-10-23) [source]

How it works

When the server finds a nearest prompt-cache prefix, it counts image starts before the cached prefix and passes only the remaining `pixel_values` through `model.embed_inputs`; generation then uses those embeddings with the KV cache. The server appends generated token IDs to the cache key before storing the updated cache. MLX-VLM separately documents that a repeated-image feature-cache hit skips its vision encoder. (medium confidence) (verified as of 2026-10-23) [source]

How-to and procedures

To distinguish images that expand to the same image-token pattern, `mlx_lm.server` replaces each contiguous image-token run in the prompt-cache key with a negative integer derived from the first 15 hexadecimal digits of SHA-256 over the inlined image data URL. MLX-VLM documents a separate `VisionFeatureCache` that keys projected vision features by image path. (medium confidence) (verified as of 2026-10-23) [source]

Comparisons and alternatives

For image requests, mlx_lm.server builds a modified cache key by replacing each run of image patch-token IDs with a negative ID derived from SHA-256 of the URI string passed to _make_vision. The HTTP handler first rewrites remote URLs as data URLs, so the hash input for remote images is the normalized data URI rather than the remote host path. The server performs image decoding and processor work before looking up the nearest cached prefix; after a hit, it calls model.embed_inputs only for images whose patch starts fall after the cached prefix. Thus this path can reuse KV and visual embedding work for cached image blocks, while still repeating earlier image input preparation on later requests. (low confidence) (verified as of 2026-10-23) [source]
ResponseGenerator._generate explicitly makes a request nonbatchable when _image_parts(request.messages) finds image content, even if other batching conditions hold. It drains any incompatible active batch, then routes that request through _serve_single; the pinned native image path is sequential rather than an image-capable BatchGenerator path. (low confidence) (verified as of 2026-10-23) [source]

Facts and statements

The standalone mlx_lm.cache_prompt CLI builds a cache from one --prompt string or stdin: it applies the tokenizer chat template or encodes the string, runs generate_step with a KV cache, and saves that state. It does not call an image processor or pass pixel_values, so this disk-cache CLI should not be assumed to store image prompt processing state. (medium confidence) (verified as of 2026-10-23) [source]
Image reuse across VLM integrations needs more than token-prefix KV matching. Current LM Studio batched vision code restores prompt caches using prepared image_spans and rebuilds cached prompt kwargs. A mlx-vlm issue describing Qwen3.5 support says image checkpoint reuse required image_end_index and get_partial_input_embeddings so the engine could resume after an image block and encode only new images. A Rapid-MLX report says its engine reran the vision tower until it forwarded mlx-vlm's vision_cache and _image_key contract, which the report says applies only to model families that implement it. Taken together, these sources indicate that a text or KV-prefix hit alone does not prove that every engine/model path skipped vision encoding. (medium confidence) (verified as of 2026-10-23) [source]
Before prompt tokenization, `mlx_lm.server` fetches HTTP(S) image URLs and rewrites them as base64 data URLs; it decodes image bytes to RGB arrays and passes those images with rendered chat text through the model processor. The OpenAI API reference documents image URLs and base64 image data as valid `image_url.url` values. (high confidence) (verified as of 2026-10-23) [source]
Image-cache behavior is serving-backend-specific: LM Studio's batched vision backend initializes a `VlmPromptCacheStore`, a prompt-cache coordinator, and a `VisionFeatureMemoizer`. An April 2026 issue reported that an earlier VisionAddOn/Qwen3.5 path re-prefilled every turn and that `LRUPromptCache` missed some hybrid Qwen3.6 system-plus-user prefixes. (medium confidence) [source]
Muse Spark Processor.__call__ splits rendered chat text on <|patch|> and requires exactly one placeholder per supplied image; a mismatch raises ValueError with the placeholder and image counts. preprocess_image resizes/normalizes each RGB image and returns channels-first pixels plus its token-grid size. Each placeholder becomes <|image_start|>, that many <|patch|> tokens, and <|image_end|>. (low confidence) (verified as of 2026-10-23) [source]
In _serve_single, image-aware prefix reuse counts vision.starts positions before ctx.prompt_cache_count and slices vision.pixel_values[n:] accordingly. Remaining pixels go through model.embed_inputs(mx.array(rest), pixel_values=...), the embeddings are evaluated, and stream_generate receives them as input_embeddings alongside prompt_cache=cache and the uncached prompt suffix. stream_generate forwards its remaining options to generate_step for the ordinary generation path. (medium confidence) (verified as of 2026-10-23) [source]

Children

Image-part normalization and model processor inputs (frontier)
URI-derived image identity in prefix cache keys (frontier)
Visual feature caches versus language-model KV caches (frontier)

Frontier under this node: Image-part normalization and model processor inputs, URI-derived image identity in prefix cache keys, Visual feature caches versus language-model KV caches

← the whole tree · 3D view· how to read this page