On-Device & Local LLM Runtimes

On-Device & Local LLM Runtimes

The local/on-device runtime + developer-experience layer: which runtime to install and how to run, serve, and size an LLM on a laptop, desktop, phone, browser, or edge device — without a cloud API. This is the “get a model talking on localhost” skill, not the quant-math skill and not the datacenter-serving skill.

When to use / Skip

Use this skill when you are:

Skip — defer to the right neighbor:

Why run locally

Driver What it buys you
Privacy / data residency Prompts + documents never leave the device. The whole premise of Apple Foundation Models, Chrome Built-in AI, Windows Foundry Local.
Cost No per-token billing — viable for high-volume, batch, or background tasks.
Offline Works on a plane, air-gapped network, or the edge.
Latency No network round-trip; on-device TTFT ~100 ms for a 3B on a modern NPU.
Control No rate limits, no surprise deprecations, reproducible runs.

The tradeoff: local models are smaller/slower per token than frontier cloud models, and you own the ops (download, RAM budget, updates). Pick local when privacy/cost/offline matters more than peak capability.

Desktop/server runtime landscape

Ollama — the default on-ramp

llama.cpp — the engine under everything

LM Studio — GUI + SDK + MLX backend

Apple MLX / MLX-LM — Apple Silicon native

MLC-LLM — universal deploy via ML compilation

Long tail (one line each)

Browser & edge runtimes

Mobile runtimes

Hardware sizing & quant-level selection

Total memory ≈ model weights + KV cache + ~0.5-1 GB runtime overhead.

Weights = params(B) × bytes-per-weight, set by the quant level:

Level ≈ bytes/wt 7B 13B 70B Quality vs FP16 Use when
FP16 2.0 ~14 GB ~26 GB ~140 GB baseline training / max fidelity
Q8_0 ~1.0 ~8 GB ~14 GB ~70 GB ~99% (near-lossless) quality-critical, RAM to spare
Q6_K ~0.75 ~6 GB ~11 GB ~54 GB ~97% reasoning/code, 20+ GB
Q5_K_M ~0.65 ~5 GB ~9 GB ~47 GB ~97-98% 12-16 GB sweet spot
Q4_K_M ~0.5-0.55 ~4-5 GB ~8 GB ~40 GB ~92-95% default; 8-10 GB chat
Q3_K_M ~0.4 ~3.5 GB ~6.5 GB ~33 GB noticeably worse only if desperate

Quick formula: memory_GB ≈ params_B × bytes_per_weight × 1.2. Selection rule (in priority order): (1) fit your RAM/VRAM ceiling, (2) quality, (3) speed. Q4_K_M is the community default; bump to Q5/Q6 for code & reasoning (they punish aggressive quant); Q8_0 when fidelity matters and it fits. Avoid Q2. (How k-quants pack bits / imatrix calibration -> llm-compression.)

KV cache — the silent long-context killer. It grows linearly with context. A 70B at 4K ctx ≈ ~2 GB KV; at 128K ctx ≈ ~64 GB for the cache alone before weights. Mitigate with KV-cache quantization (--cache-type-k q8_0) and GQA (Llama 3.1 8B: 8 KV heads vs 32 query heads -> ~4x smaller cache).

VRAM-tier cheat sheet (Q4_K_M, ~4K ctx):

VRAM/RAM Comfortably runs
8 GB 7-8B (RTX 3070/4060, many laptops)
12 GB 13B (RTX 3060 12 GB)
24 GB 32-34B; 70B only with partial CPU offload (RTX 3090/4090)
48 GB 70B Q4 (RTX 6000 Ada / dual 24 GB)
128 GB+ unified (Mac) 70B+ at higher quant; very large MoE

When weights exceed VRAM, use partial offload: llama.cpp --n-gpu-layers N (GPU layers, rest on CPU) — graceful but slower.

Apple Silicon vs consumer NVIDIA (the durable frame). Two philosophies:

Takeaway: NVIDIA for speed at sizes that fit; Apple to run models that don’t fit a consumer GPU. Indicative single-stream decode: dense 70B-Q4 ~8-15 tok/s on an M4 Max (faster at short context). Treat headline “2,000+ tok/s” figures with care — those are MoE (few active params) and/or prefill/batched, not dense decode. For multi-user throughput on one box you eventually outgrow these single-stream runtimes -> llm-inference-serving.

Integration patterns

Point any OpenAI client at a local server — same SDK, local base_url, dummy key:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")  # or :8080 (mlx/llama.cpp), :1234 (LM Studio)
resp = client.chat.completions.create(
    model="llama3.1", messages=[{"role": "user", "content": "Hi"}])

Structured output (Ollama, JSON schema):

from ollama import chat
from pydantic import BaseModel
class Country(BaseModel):
    name: str; capital: str
out = chat(model="llama3.1",
           messages=[{"role": "user", "content": "Tell me about Canada."}],
           format=Country.model_json_schema(), options={"temperature": 0})
Country.model_validate_json(out.message.content)

Grammar-constrained output (llama.cpp GBNF): pass a .gbnf to llama-server’s grammar param to force, e.g., valid JSON or a fixed enum — useful when a model lacks native structured-output support.

Local embeddings:

curl http://localhost:11434/v1/embeddings -d '{"model":"nomic-embed-text","input":"hello"}'
# llama.cpp:  llama-server -m embed.gguf --embeddings   ->  POST /v1/embeddings

In-browser (WebLLM):

import { CreateMLCEngine } from "@mlc-ai/web-llm";
const engine = await CreateMLCEngine("Llama-3.2-3B-Instruct-q4f16_1-MLC");
const r = await engine.chat.completions.create({ messages:[{role:"user",content:"Hi"}] });

Apple guided generation (Swift):

@Generable struct Recipe { let title: String; let steps: [String] }
let session = LanguageModelSession()
let recipe = try await session.respond(to: "A quick pasta recipe", generating: Recipe.self)

Anti-patterns & failure modes

2025-2026 frontier

Sources

  1. Ollama — OpenAI compatibility / Modelfile / structured outputs / API / multimodal: https://docs.ollama.com/api/openai-compatibility , /modelfile , /capabilities/structured-outputs , https://github.com/ollama/ollama/blob/main/docs/api.md , https://ollama.com/blog/multimodal-models
  2. llama.cpp — llama-server README: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
  3. LM Studio — Local server docs + MLX (v0.3.4): https://lmstudio.ai/docs/developer/core/server , https://lmstudio.ai/blog/lmstudio-v0.3.4
  4. Apple MLX-LM: https://github.com/ml-explore/mlx-lm
  5. MLC-LLM + WebLLM: https://llm.mlc.ai/docs/get_started/introduction , https://github.com/mlc-ai/web-llm
  6. Chrome Built-in AI / Prompt API + I/O ’25 status: https://developer.chrome.com/docs/ai/prompt-api , https://developer.chrome.com/blog/ai-api-updates-io25
  7. Apple Foundation Models framework + 2025 updates: https://developer.apple.com/documentation/FoundationModels , https://machinelearning.apple.com/research/apple-foundation-models-2025-updates
  8. Microsoft Foundry on Windows + Windows ML GA: https://learn.microsoft.com/en-us/windows/ai/overview , https://blogs.windows.com/windowsdeveloper/2025/09/23/windows-ml-is-generally-available…
  9. Google AI Edge — MediaPipe LLM Inference (LiteRT-LM migration): https://ai.google.dev/edge/mediapipe/solutions/genai/llm_inference
  10. Microsoft ONNX Runtime GenAI + Snapdragon/QNN: https://github.com/microsoft/onnxruntime-genai , https://onnxruntime.ai/docs/genai/tutorials/snapdragon.html
  11. Mozilla llamafile: https://github.com/mozilla-ai/llamafile
  12. GGUF VRAM/memory calculators + quant guide (sizing corroboration): https://ggufvram.radicchio.page/ , https://llmhardware.io/guides/llm-quantization-guide
  13. Apple Silicon vs RTX local-LLM benchmarks: https://www.sitepoint.com/mac-m3-max-vs-rtx-4090-local-llm-benchmark/ , https://github.com/XiongjieDai/GPU-Benchmarks-on-LLM-Inference

Boundary note: quant-algorithm/format internals (GPTQ/AWQ, GGUF k-quant math, imatrix) defer to llm-compression; datacenter/multi-GPU serving (vLLM, batching, disaggregation) to llm-inference-serving. Quant levels (Q4/Q5/Q8) appear here only as a fit/quality knob. Not related: dexie-indexeddb (browser storage) and mongodb-atlas-device-sdk (Realm sync) — different domains despite “local/device” keyword overlap.