On-Device & Local LLM Runtimes

Parent: LLM Models and APIs · researched 2026-06-03T22:51:20.748Z· 18 sources · 12 concepts · skill on-device-local-llm-runtimes

The local/on-device runtime + developer-experience layer: which runtime to

On-Device & Local LLM Runtimes

The local/on-device runtime + developer-experience layer: which runtime to [source]
install and how to run, serve, and size an LLM on a laptop, desktop, phone, [source]
browser, or edge device - without a cloud API. This is the "get a model talking [source]
on localhost" skill, not the quant-math skill and not the datacenter-serving [source]

When to use / Skip

Use this skill when you are: [source]
Choosing a local runtime (Ollama vs llama.cpp vs LM Studio vs MLX vs MLC vs [source]
Jan vs GPT4All vs llamafile vs KoboldCpp). [source]
Sizing hardware: "how much RAM/VRAM for a 7B/13B/70B?", "which quant fits 8/12/24 GB?". [source]
Standing up a local OpenAI-compatible server and pointing app code at it. [source]
Running a model in the browser (WebGPU / WebLLM / Transformers.js / Chrome [source]
Prompt API) or on a phone (MediaPipe/LiteRT-LM / ONNX Runtime GenAI / [source]
llama.cpp), or via OS frameworks (Apple Foundation Models, Windows Foundry Local). [source]
Getting local embeddings or local structured output / function calling. [source]
Skip - defer to the right neighbor: [source]
Quantization ALGORITHMS & format internals - GPTQ/AWQ, GGUF k-quant math, [source]
imatrix, bitsandbytes/NF4 -> llm-compression. (Quant levels as a [source]
fit/quality/speed knob are HERE; why Q4_K_M is 4.8 bpw is THERE.) [source]
Datacenter / server-grade serving - vLLM PagedAttention, continuous/in-flight [source]
batching, prefill/decode disaggregation, TTFT/TPOT goodput SLOs, multi-GPU [source]
tensor/pipeline parallelism -> llm-inference-serving. (Single-machine local [source]
serving is HERE; the high-throughput engine is THERE.) [source]
Consuming a hosted cloud model / managed endpoint -> llm-integration-reviewer, [source]
aws-ai-ml. Picking the best model on quality/price -> llm-models. [source]

Why run locally

The tradeoff: local models are smaller/slower per token than frontier cloud [source]
models, and you own the ops (download, RAM budget, updates). Pick local when [source]
privacy/cost/offline matters more than peak capability. [source]

Ollama — the default on-ramp

Go daemon over a llama.cpp-derived engine. ollama run llama3.1 pulls from the [source]
model library and serves on localhost:11434. [source]
APIs: OpenAI-compatible /v1/chat/completions, /v1/completions, [source]
/v1/embeddings, /v1/responses (non-stateful), tools/function-calling, JSON [source]
mode; plus native /api/chat, /api/generate, /api/embed. [source]
Modelfile = build blueprint: FROM, PARAMETER (e.g. num_ctx 8192, [source]
temperature), SYSTEM, TEMPLATE (Go template), ADAPTER (LoRA). Build with [source]
ollama create mymodel -f Modelfile. Setting context size for the OpenAI API [source]
requires a Modelfile with PARAMETER num_ctx N (the /v1 API can't set it). [source]
Structured outputs (since late 2024): pass a JSON schema to format [source]
(use Pydantic model_json_schema() / Zod zodToJsonSchema(); set temperature 0). [source]
Tool calling with Llama 3.1/3.2, Mistral, Qwen2.5, gpt-oss; streaming tool calls. [source]
2025: new multimodal engine (vision first-class); native desktop GUI app [source]
(macOS/Windows) shipped 2025-07-30; drag-drop PDFs/images. [source]

llama.cpp — the engine under everything

C/C++ inference engine; powers Ollama, LM Studio, llamafile, KoboldCpp. [source]
Consumes GGUF (the dominant local quant format; 100k+ GGUF repos on HF). [source]
llama-server exposes POST /v1/chat/completions, /v1/completions, [source]
/v1/embeddings, Anthropic-style /v1/messages (with --jinja), and a built-in [source]
web UI (--no-ui to disable). [source]
Constrained generation: GBNF grammars via the grammar param; [source]
json_schema + response_format on the chat endpoint. [source]
Key flags: --model, --ctx-size N, --n-gpu-layers N (offload N layers to [source]
GPU/Metal - the central partial-offload knob), --host, --port, --embeddings, [source]
--cache-type-k q8_0 (quantize the KV cache). Python wrapper: llama-cpp-python [source]
ships the same OpenAI server. [source]

LM Studio — GUI + SDK + MLX backend

Polished desktop app to discover/download/run models; **local OpenAI-compatible [source]
server** (Developer tab, REST + lms CLI + TS/Python SDKs). [source]
Ships an Apple MLX backend (since v0.3.4) - MLX-accelerated inference on Mac [source]
behind the same UI/API. JSON-schema structured output supported. [source]

Apple MLX / MLX-LM — Apple Silicon native

MLX = Apple's array framework exploiting unified memory (CPU+GPU share one pool). [source]
mlx-lm Python pkg: mlx_lm.generate, mlx_lm.chat, [source]
mlx_lm.convert --model … -q (quantize, e.g. 4-bit), and [source]
mlx_lm.server --model … --port 8080 -> OpenAI-compatible /v1. [source]
mlx-community on HF hosts thousands of pre-converted models. Mac-only [source]
(M-series); not Intel Macs. [source]

MLC-LLM — universal deploy via ML compilation

Compiler (Apache TVM Unity) -> MLCEngine, an OpenAI-compatible engine across [source]
REST / Python / JS / iOS / Android. [source]
mlc_llm compile … -o model.wasm for the browser; native libs for phones. [source]
WebLLM is its web backend. Use when you need one model artifact deployed to [source]

Long tail (one line each)

llamafile (Mozilla): llama.cpp + Cosmopolitan Libc -> a single [source]
Actually-Portable-Executable running on 6 OSes, no install, weights embedded. [source]
GPT4All (Nomic): zero-setup desktop app, CPU-friendly, LocalDocs [source]
Jan: open-source "ChatGPT replacement" UI with hybrid local/cloud switching. [source]
KoboldCpp: llama.cpp fork + KoboldAI UI + OpenAI-compatible API; story/roleplay focus. [source]
Open WebUI / AnythingLLM / LocalAI: front-ends/gateways on top of Ollama or any [source]
/v1 server (RAG UI, multi-backend). LocalAI also fronts multiple backends/formats. [source]

Browser & edge runtimes

WebLLM (MLC): high-performance in-browser engine, WebGPU-accelerated, [source]
OpenAI-compatible JS API (CreateMLCEngine), Web Worker support, runs fully [source]
client-side. npm i @mlc-ai/web-llm or CDN. Best for local chat with streaming. [source]
Transformers.js v3 (HuggingFace): Python-Transformers API in JS over **ONNX [source]
Runtime Web -> WebGPU** (3-10x the WASM fallback). Best for HF pipelines and [source]
in-browser embeddings / RAG. [source]
Chrome Built-in AI / Gemini Nano (Prompt API): the browser ships Gemini Nano. [source]
Prompt API is stable for web pages as of Chrome 148 (extensions-only through [source]
~138); Summarizer stable since 138; Translator + Language Detector [source]
stable in 148; Writer/Rewriter in origin trial; multimodal input in the Early [source]
Preview Program. Full GA targeted ~Chrome 150 / end of 2026. No model download for [source]
the developer - the browser manages weights. [source]
Apple Foundation Models framework (WWDC25, iOS/macOS 26): Swift API to Apple [source]
Intelligence's ~3B on-device model (KV-cache sharing + 2-bit QAT). **Guided [source]
generation = constrained decoding via the @Generable** macro on Swift [source]
structs/enums (OS daemon runs constrained + speculative decoding); plus **tool [source]
calling**. On-device, private, free to the app. [source]
Windows AI Foundry / Foundry Local (Build 2025; evolution of Windows Copilot [source]
Runtime): Windows ML (GA 2025-09) = on-device runtime across CPU/GPU/NPU [source]
(AMD/Intel/NVIDIA/Qualcomm). Foundry Local auto-detects hardware, lists [source]
compatible models, and exposes an OpenAI-compatible local endpoint; ships [source]
Phi-class SLMs (~3.8B) tuned for Copilot+ PC NPUs. [source]

Mobile runtimes

llama.cpp on iOS/Android: ARM + Metal builds power most third-party local-LLM [source]
apps (e.g. PocketPal); GGUF models via Swift/Kotlin wrappers. [source]
Google MediaPipe LLM Inference: cross-platform on-device API (Gemma 3n [source]
E2B/E4B, Phi-2…) using .task files; multimodal on Android. **2025 change: the [source]
Android/iOS implementations are DEPRECATED -> migrate to LiteRT-LM. The Web** [source]
target is not deprecated. [source]
ONNX Runtime GenAI (Microsoft): generative layer over ONNX Runtime; execution [source]
providers CPU/CUDA/DirectML/QNN/OpenVINO/WebGPU. Runs Phi-3.5-mini and [source]
Llama-3.2-3B on Qualcomm NPUs (PC + mobile); ~100 ms TTFT (128-token prompt, [source]
Snapdragon 8 Elite). Pairs with Qualcomm AI Hub for precompiled QNN binaries. [source]

Hardware sizing & quant-level selection

Total memory ≈ model weights + KV cache + ~0.5-1 GB runtime overhead. [source]
Weights = params(B) × bytes-per-weight, set by the quant level: [source]
Quick formula: memory_GB ≈ params_B × bytes_per_weight × 1.2. [source]
Selection rule (in priority order): **(1) fit your RAM/VRAM ceiling, (2) quality, [source]
(3) speed.** Q4_K_M is the community default; bump to Q5/Q6 for code & reasoning [source]
(they punish aggressive quant); Q8_0 when fidelity matters and it fits. Avoid Q2. [source]
KV cache - the silent long-context killer. It grows linearly with context. [source]
A 70B at 4K ctx ≈ ~2 GB KV; at 128K ctx ≈ ~64 GB for the cache alone before [source]
weights. Mitigate with KV-cache quantization (--cache-type-k q8_0) and [source]
GQA (Llama 3.1 8B: 8 KV heads vs 32 query heads -> ~4x smaller cache). [source]
VRAM-tier cheat sheet (Q4_K_M, ~4K ctx): [source]
When weights exceed VRAM, use partial offload: llama.cpp --n-gpu-layers N [source]
(GPU layers, rest on CPU) - graceful but slower. [source]
Apple Silicon vs consumer NVIDIA (the durable frame). Two philosophies: [source]
Apple unified memory - one large pool (up to 128-512 GB on Max/Ultra), but [source]
bandwidth-bound (~546 GB/s on M4 Max). *Lets large models load that a 24 GB [source]
card can't hold at all.* [source]
NVIDIA discrete VRAM - faster (~1008 GB/s on RTX 4090), **~2x+ faster per [source]
token when the model fits, but 24 GB is a hard wall** - spill to system RAM [source]
and throughput collapses. [source]
**Takeaway: NVIDIA for speed at sizes that fit; Apple to run models that don't [source]
fit a consumer GPU. Indicative single-stream decode: dense 70B-Q4 ~8-15 tok/s [source]
on an M4 Max** (faster at short context). Treat headline "2,000+ tok/s" figures [source]
with care - those are MoE (few active params) and/or prefill/batched, not dense [source]
decode. For multi-user throughput on one box you eventually outgrow these [source]
single-stream runtimes -> llm-inference-serving. [source]

Integration patterns

Point any OpenAI client at a local server - same SDK, local base_url, dummy key: [source]
Structured output (Ollama, JSON schema): [source]
Grammar-constrained output (llama.cpp GBNF): pass a .gbnf to llama-server's [source]
grammar param to force, e.g., valid JSON or a fixed enum - useful when a model [source]
lacks native structured-output support. [source]
Local embeddings: [source]
In-browser (WebLLM): [source]
Apple guided generation (Swift): [source]

Anti-patterns & failure modes

Picking quant by name, not by fit. Always size first (params × bytes × 1.2 [source]
+ KV). A 70B-Q4 (~40 GB) will not fit a 24 GB card without offload. [source]
Forgetting the KV cache at long context. Long-context jobs OOM from the cache, [source]
not the weights - quantize the KV cache or cut context. [source]
Defaulting to Q4 for code/reasoning. Those workloads degrade visibly; prefer [source]
Q5/Q6 if you have headroom. [source]
Setting context on Ollama's /v1 API and wondering why it's ignored. Bake [source]
num_ctx into a Modelfile; the OpenAI-compat layer can't set it. [source]
Exposing a local server beyond loopback. llama.cpp/Ollama default to localhost; [source]
binding --host 0.0.0.0 puts an unauthenticated LLM on your LAN. Gate it. [source]
Reaching for a local single-stream runtime to serve many users. Concurrency, [source]
batching, autoscaling -> llm-inference-serving (vLLM/SGLang/TGI), not Ollama. [source]
Assuming WebGPU/Built-in AI everywhere. WebGPU + Chrome Built-in AI gate on [source]
browser version, OS, and hardware; always feature-detect and fall back. [source]
Shipping a now-deprecated mobile path. MediaPipe LLM on Android/iOS is [source]
deprecated -> LiteRT-LM; don't start new mobile work on it. [source]
Trusting unconditional throughput numbers. Verify model (dense vs MoE), [source]
context length, and decode-vs-prefill before quoting tok/s. [source]

2025-2026 frontier

OpenAI-compat is the lingua franca. Ollama, llama.cpp, LM Studio, MLX-LM, [source]
MLC, Foundry Local all expose /v1 - local<->cloud swap is a base_url change. [source]
NPUs go mainstream. Copilot+ PCs (Windows ML/Foundry Local), Snapdragon [source]
(ONNX Runtime GenAI + QNN), and Apple's Neural-Engine-assisted stack push small [source]
models onto dedicated low-power silicon. [source]
OS-native on-device models. Apple Foundation Models (~3B, @Generable), [source]
Chrome Gemini Nano (Prompt API), Windows Phi-Silicon - "free," private, zero-download [source]
models built into the platform. [source]
Small-but-capable models. Gemma 3n (E2B/E4B), Llama 3.2 1B/3B, Qwen2.5, [source]
Phi-class, gpt-oss - the 1-4B tier is now genuinely useful on-device. [source]
Local serving borrows datacenter tricks. Projects like vLLM-MLX bring [source]
continuous batching / paged-KV to Apple Silicon - when you outgrow single-stream [source]
local serving, that's the bridge to llm-inference-serving. [source]
Browser inference matures. WebGPU is broadly shipping; WebLLM + Transformers.js [source]
make zero-install, fully-private web AI practical for sub-4B models. [source]

Sources

Ollama - OpenAI compatibility / Modelfile / structured outputs / API / multimodal: https://docs.ollama.com/api/openai-compatibility , /modelfile , /capabilities/structured-outputs , https://github.com/ollama/ollama/blob/main/docs/api.md , https://ollama.com/blog/multimodal-models [source]
llama.cpp - llama-server README: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md [source]
LM Studio - Local server docs + MLX (v0.3.4): https://lmstudio.ai/docs/developer/core/server , https://lmstudio.ai/blog/lmstudio-v0.3.4 [source]
Apple MLX-LM: https://github.com/ml-explore/mlx-lm [source]
MLC-LLM + WebLLM: https://llm.mlc.ai/docs/get_started/introduction , https://github.com/mlc-ai/web-llm [source]
Chrome Built-in AI / Prompt API + I/O '25 status: https://developer.chrome.com/docs/ai/prompt-api , https://developer.chrome.com/blog/ai-api-updates-io25 [source]
Apple Foundation Models framework + 2025 updates: https://developer.apple.com/documentation/FoundationModels , https://machinelearning.apple.com/research/apple-foundation-models-2025-updates [source]
Microsoft Foundry on Windows + Windows ML GA: https://learn.microsoft.com/en-us/windows/ai/overview , https://blogs.windows.com/windowsdeveloper/2025/09/23/windows-ml-is-generally-available... [source]
Google AI Edge - MediaPipe LLM Inference (LiteRT-LM migration): https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference [source]
Microsoft ONNX Runtime GenAI + Snapdragon/QNN: https://github.com/microsoft/onnxruntime-genai , https://onnxruntime.ai/docs/genai/tutorials/snapdragon.html [source]
Mozilla llamafile: https://github.com/mozilla-ai/llamafile [source]
GGUF VRAM/memory calculators + quant guide (sizing corroboration): https://ggufvram.radicchio.page/ , https://llmhardware.io/guides/llm-quantization-guide [source]
Apple Silicon vs RTX local-LLM benchmarks: https://www.sitepoint.com/mac-m3-max-vs-rtx-4090-local-llm-benchmark/ , https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference [source]
> Boundary note: quant-algorithm/format internals (GPTQ/AWQ, GGUF k-quant math, imatrix) defer to llm-compression; datacenter/multi-GPU serving (vLLM, batching, disaggregation) to llm-inference-serving. Quant levels (Q4/Q5/Q8) appear here only as a fit/quality knob. Not related: dexie-indexeddb (browser storage) and mongodb-atlas-device-sdk (Realm sync) - different domains despite "local/device" keyword overlap. [source]

Children

Ollama Modelfile & local model library (frontier)
llama.cpp / llama-server & the GGUF format (frontier)
GBNF grammars & local schema-constrained decoding (frontier)
Apple MLX & MLX-LM (unified-memory inference) (frontier)
LM Studio (GUI + SDK + MLX backend) (frontier)
MLC-LLM & MLCEngine (universal/compiled deploy) (frontier)
WebLLM & Transformers.js (WebGPU in-browser inference) (frontier)
Chrome Built-in AI / Gemini Nano Prompt API (frontier)
Apple Foundation Models framework & @Generable guided generation (frontier)
Windows AI Foundry / Foundry Local & on-device NPU inference (frontier)
Mobile on-device runtimes (MediaPipe to LiteRT-LM, ONNX Runtime GenAI / QNN) (frontier)
Local hardware sizing & quant-level selection (RAM/VRAM, KV cache, Q4/Q5/Q8) (frontier)

Frontier under this node: Apple Foundation Models framework & @Generable guided generation, Apple MLX & MLX-LM (unified-memory inference), Chrome Built-in AI / Gemini Nano Prompt API, GBNF grammars & local schema-constrained decoding, LM Studio (GUI + SDK + MLX backend), Local hardware sizing & quant-level selection (RAM/VRAM, KV cache, Q4/Q5/Q8), MLC-LLM & MLCEngine (universal/compiled deploy), Mobile on-device runtimes (MediaPipe to LiteRT-LM, ONNX Runtime GenAI / QNN), Ollama Modelfile & local model library, WebLLM & Transformers.js (WebGPU in-browser inference), Windows AI Foundry / Foundry Local & on-device NPU inference, llama.cpp / llama-server & the GGUF format

← the whole tree · 3D view· how to read this page