Mac local LLMs: Apple Foundation Models and Core AI

Parent: Running LLM models locally on a Mac · Published reference · snapshot 2026-10-05

macOS 27 shipped 2026-09-14. SystemLanguageModel exists from macOS 26.0 (versions 26.0-26.3, 26.4, 27.0); 26.4 added `contextSize` and `tokenCount(for:)`.

These notes link each claim to its source. A source may be a research report hosted on this site rather than the primary document. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

System model, versions, context

macOS 27 shipped 2026-09-14. SystemLanguageModel exists from macOS 26.0 (versions 26.0-26.3, 26.4, 27.0); 26.4 added `contextSize` and `tokenCount(for:)`. [source]
On-device context is disputed: session 241 prints 8192; session 319 says "4k" vs 32K PCC; TN3193 says 4096. Read `SystemLanguageModel.default.contextSize`; overflow throws. [source]
On-device: offline, no request limits. PCC: needs network, daily per-user limit, free under 2M first-time downloads with an entitlement. [source]
Adapter toolkit ended at 26.0.0 (not OS 27 compatible); adapters ~160 MB. [source]

fm CLI and fm serve

/usr/bin/fm refuses until `sudo fm license` ("YOU HAVE NOT AGREED TO THE FOUNDATION MODELS CLI LEGAL NOTICE & TERMS"); applies to all users. [source]
Subcommands: respond, chat, schema, count-tokens, available, license, serve. `respond`: `-i`, `--text`, `--schema`, `-g`, `--no-stream`, `--save-transcript`, `--resume`, `--image`, `--tool` (barcode, ocr). [source]
`fm serve`: 127.0.0.1:1976, /health, /v1/models, /v1/chat/completions, model id `system`. Recaps citing localhost:8000 and "apple-fm" were wrong. [source]
Not OpenAI drop-in: streams by default (send `"stream": false`); `tool_choice: "auto"` puts raw JSON in `content`, no `tool_calls`; `"required"` gives HTTP 500. [source]
No auth (any bearer token returns 200); only 403 "Cross-site requests are not allowed"; Apple's help example binds 0.0.0.0. `--socket` is srwxr-xr-x, so use a 0700 directory. [source]
Schema is shape only: injection followed 20/20; missing `age` became 30; 8/60 runs hit a context-size error; `--save-transcript` is 0644. [source]
Python: `pip install apple-fm-sdk`, `import apple_fm_sdk as fm`; macOS 26.0+, matching Xcode, Python 3.10+. Wrong Xcode gives "model not available". [source]

LanguageModel protocol and Dynamic Profiles

Executors share KV cache per equal Hashable configuration, get the full transcript each call and must diff it. `prewarm` may not run; load lazily. [source]
Appending keeps the KV cache; removing entries, changing tools or editing instructions invalidates it. [source]
`historyTransform` alters only what the model sees (example `history.suffix(20)`); hitting maximumResponseTokens stops without throwing; switching remote/on-device loses the cache. [source]

mlx-swift-lm adapter and AnyLanguageModel

mlx-swift-lm 3.32.3 (2026-09-30) first includes MLXFoundationModels: `.upToNextMinor(from: "3.32.3")`, swift-huggingface 0.9.0, swift-transformers 1.3.0. Floors stay macOS 14; adapter is `@available(macOS 27.0)`. [source]
Use `#huggingFaceLanguageModel(configuration: LLMRegistry.qwen3_0_6b_4bit, capabilities: [.reasoning])`; default capability `.guidedGeneration`. On the 26 SDK the trait `FoundationModelsIntegration` compiles to an empty module. [source]
Guided generation masks logits (JSON Schema, EBNF, structural tags). First call per schema blocks hundreds of ms: avoid `@MainActor`, pre-warm a throwaway `GrammarConstraint`. [source]
Issues: #543 (no compile on Xcode 27 beta 5; final status unverified); #539 (27-SDK test bundle crashes on macOS 26 host). SwiftPM 9286: "exhausted attempts to resolve the dependencies graph"; add trait packages directly. [source]
AnyLanguageModel 0.15.1 (2026-10-02) is maintained, not pre-27: Swift 6.3+, traits CoreML/MLX/Llama, Ollama at http://localhost:11434, `MLXLanguageModel(modelId:)`; MLX tests need `xcodebuild`. [source]
Rule: mlx-swift-lm `ChatSession` for macOS 14+; MLXFoundationModels on the 27 SDK; AnyLanguageModel for macOS 14-26 or llama.cpp/Ollama. [source]

Core AI

`uv run coreai.llm.export <HF_ID> [--platform iOS]`, then `CoreAILanguageModel(resourcesAt:)` in `LanguageModelSession(model:)`. Start ~0.6B; Apple ships recipes only. Needs OS and Xcode 27+. [source]
macOS export 4bit INT4 with dynamic KV; iOS needs `--max-context-length`. Engine (GPU/CPU/ANE) depends on export. AOT: `xcrun coreai-build compile`; builds with `.aimodel` fail without the Metal Toolchain; caches reset on OS update. [source]
Trap: same Qwen3-0.6B export ran 1,121 tok/s on macOS 26 but 484 on a macOS 27 beta; re-export does not fix. gpt-oss-20B export needed 128 GB. [source]
M4 Max, one author, Core AI vs MLX: gpt-oss-20b 78.1 vs 100.2; qwen3-8b 90.0 vs 94.1; qwen3-4b tie. Prefer MLX on Mac for HF, MoE, hybrids. [source]

Utilities package and prefix cache

`ChatCompletionsLanguageModel(name:url:additionalHeaders:supportsGuidedGeneration:urlSessionConfiguration:)`; `v<digits>` path part gets `/chat/completions`, else `/v1/chat/completions`; README's `http://localhost/v1:8000` is malformed. Set `supportsGuidedGeneration: false` if `response_format` is ignored. [source]
Always sends `stream_options`, `max_completion_tokens`; rejecting servers return HTTP 400. Seed throws `RequestError.invalidRequest`. [source]
`summarizeHistory` counts entries, not tokens; pair with `rollingWindow`. Head rewrites defeat server prefix caches; prompt-based skills append only. [source]
llama-server `n_cache_reuse` shifts matching chunks, else logs "cache reuse is not supported". oMLX hashes chain from the head. Claude Code deleting old tool results forces re-prefill; `CLAUDE_AUTOCOMPACT_PCT_OVERRIDE=95` worsens it. [source]

Corrections and open questions

Fixed: shipping fm 2.0.68.1.402 lists only `system` (no PCC); Python SDK is macOS 26.0+; Core AI is not ANE-only. [source]
Unverified: 3.32.3 on final Xcode 27.x; `fm serve` as ChatCompletions backend; Anthropic/Google packages shipped. [source]

Corrections and disagreements

CONTRADICTS: on-device-local-llm-runtimes.md says fm CLI offers PCC; the shipping macOS 27.0 build (fm 2.0.68.1.402) tested lists only the `system` model while WWDC and a beta write-up show `--model pcc` and `quota-usage`. [source]
CONTRADICTS: on-device-local-llm-runtimes.md says OS 27 on-device context is 8192; session 319 says the on-device model offers "4k" context vs 32K for PCC, and TN3193 says 4096 per session. [source]
CONTRADICTS (within sources): articles show `MLXLanguageModel(modelID: "mlx-community/...")`, but the adapter README documents a macro or `configuration:`/`weightsLocation:`/`load:`; the session sample's `modelID:` is Apple's placeholder. [source]
CONTRADICTS: on-device-local-llm-runtimes.md lists the Python SDK under OS 27; the SDK requires macOS 26.0+ and has existed since 2026-02. [source]
CONTRADICTS (soft): on-device-local-llm-runtimes.md lists "rank-32 LoRA adapters that must be retrained per model version" without noting that no adapter toolkit exists for OS 27; the toolkit ended at 26.0.0. [source]
Does CoreAILanguageModel run on the ANE or GPU? CONTRADICTS apple-foundation-models-framework.md line 96 ("CoreAILanguageModel (Neural Engine)"): Apple's article says it runs on GPU, CPU or ANE depending on export; macOS export is dynamic and lands on the GPU pipelined engine, iOS static export on the ANE. [source]
CONTRADICTS apple-foundation-models-framework.md (soft): that dossier calls AnyLanguageModel a "pre-27" package; v0.15.1 (2026-10-02) is still maintained and mirrors the 27 API (DynamicInstructions, Usage, reasoning entries) and wraps any 27 `LanguageModel` with `FoundationLanguageModel`. [source]
CONTRADICTS a macOS 27 Medium article: its Package.swift uses `.upToNextMinor(from: "1.0.0")` and `.macOS(.v27)` for mlx-swift-lm, but the repo is at 3.32.3 with macOS 14 floors. [source]
CONTRADICTS the repo's SKILL.md (stale): it describes traits ChatCompletions, Skills and History and a `.model(...)` profile call; Package.swift has no traits and beta 3 removed `.model(...)`. [source]
CONTRADICTS apple-foundation-models-framework.md on provenance only: a June 29 2026 article says Apple's WWDC session on the fm CLI shows fm respond, fm chat and fm schema and never mentions serve, so early fm serve walkthroughs were not backed by Apple's session. [source]
CONTRADICTS the early recaps: the widely shared claims of http://localhost:8000/v1/ and model id "apple-fm" for fm serve are not what the shipping build uses (127.0.0.1:1976, model system per the existing dossier). [source]

Concepts in this cluster

Apple Foundation Models framework [source]
Core AI framework [source]
mlx-swift-lm and MLXFoundationModels adapter [source]
Apple foundation-models-utilities ChatCompletionsLanguageModel [source]
MLXGuidedGeneration and XGrammar constrained decoding on Apple Silicon [source]
Server-side prefix cache behavior under rolling-window transcripts [source]
Dynamic Profiles historyTransform [source]
fm serve and the fm CLI OpenAI-compatible endpoint [source]

Children

← the whole tree · 3D view· how to read this page