Docker Model Runner

Parent: Ollama alternatives and model sourcing · Published reference · snapshot 2026-10-14 · skill ollama-expert/references/ollama-alternatives-and-model-sourcing.md

Also known as: Docker Model Runner relative to Ollama

↓ Facts as markdown all context files

Everything the scanned sources say about Docker Model Runner relative to Ollama: 68 source-anchored units across 6 sources, grouped by facet.

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

How it works

Models are pulled from Docker Hub, an OCI-compliant registry, or [Hugging Face](https://huggingface.co/) the first time you use them and are stored locally. They load into memory only at runtime when a request is made, and unload when not in use to optimize resources. Because models can be large, the initial pull may take some time. After that, they're cached locally for faster access. You can interact with the model using [OpenAI and Ollama-compatible APIs](/ai/model-runner/api-reference/). [source]
Docker Model Runner (DMR) makes it easy to manage, run, and deploy AI models using Docker. Designed for developers, Docker Model Runner streamlines the process of pulling, running, and serving large language models (LLMs) and other AI models directly from Docker Hub, any OCI-compliant registry, or [Hugging Face](https://huggingface.co/). [source]
When we first introduced Docker Model Runner, our goal was to make it simple for developers to run and experiment with large language models (LLMs) using Docker. We designed it to integrate multiple inference engines from day one, starting with llama.cpp, to make it easy to get models running anywhere. [source]
That’s because Docker Model Runner automatically routes the request to the correct inference engine based on the model you’re using, ensuring a seamless experience whether you’re using llama.cpp or vLLM. [source]
When using Docker Model Runner with Docker Engine, HEAD requests to Docker Hub are made to track model names, regardless of any settings. [source]
Docker Model Runner API: OpenAI-compatible endpoints served automatically when running models. No additional configuration needed. [source]
We'll show you how to configure both in Spring AI, how they perform under different prompt complexities, and when it might be worth switching from Ollama to Docker Model Runner, or sticking with what you have. [source]
Running large language models (LLMs) locally has become increasingly popular for privacy, cost control, and offline capabilities. The landscape shifted significantly in April 2025 when Docker introduced Docker Model Runner (DMR), its official solution for AI model deployment. [source]

How-to and procedures

With seamless integration into Docker Desktop and Docker Engine, you can serve models via OpenAI and Ollama-compatible APIs, package GGUF files as OCI Artifacts, and interact with models from both the command line and graphical interface. [source]
If your goal is simply to run an LLM inside Docker, both Docker Model Runner and Ollama in Docker containers can achieve this. [source]
Enable Docker Model Runner in Docker Desktop or install docker-model-plugin [source]

Measurements and reference values

Docker Model Runner: Performance comparable to Ollama as both support GGUF quantized models. For Llama 2 7B (Q4), expect 20-30 tokens/second on CPU and 50-80 tokens/second on mid-range GPUs. Minimal container overhead. [source]
Docker Model Runner: Similar to Ollama with automatic model loading. GGUF Q4 models typically use 4-6GB RAM. Container overhead is minimal (tens of MB). [source]
DMR serves an OpenAI-compatible API and an Ollama-compatible API. The default local endpoint is http://localhost:12434/v1/chat/completions. [source]
Update API endpoints from localhost:11434 to DMR endpoint (typically localhost:8080) [source]
dockerClient pointing to the Docker Model Runner endpoint (default: "http://localhost:12434/engines") [source]

Problems, failure modes and limitations

Practical guidance: choose DMR for Docker-native workflows, OCI artifact distribution and CI (Testcontainers); choose Ollama for rapid prototyping, framework independence and a larger one-command model library. Some CUDA images reportedly hardcode a 4096-token context limit that overrides higher settings. [source]
llama.cpp is the default engine and works on all platforms. vLLM requires NVIDIA GPUs and is supported on Linux x86_64 and Windows with WSL2. Diffusers enables image generation and requires NVIDIA GPUs on Linux (x86_64 or ARM64). See [Inference engines](/ai/model-runner/inference-engines/) for detailed comparison and setup. [source]
Docker Model Runner: Free with Docker Desktop (personal/educational) or Docker Engine. Docker Desktop requires subscription for larger organizations (250+ employees or $10M+ revenue). Models distributed through Docker Hub follow Docker’s registry pricing (free public repos, paid private repos). [source]
Context size configuration can significantly impact memory usage and model behavior. By default, some Docker Model Runner CUDA images hardcode a 4096-token limit, even if higher values are specified in docker-compose. For detailed steps on overriding this behavior and packaging models with custom context sizes, see our guide on configuring context size in Docker Model Runner. [source]
Thanks for trying out Docker Model Runner. To report bugs or request features, [open an issue on GitHub](https://github.com/docker/model-runner/issues). You can also give feedback through the **Give feedback** link next to the **Enable Docker Model Runner** setting. [source]

Comparisons and alternatives

Tool Calling Support: Docker Model Runner’s tool calling capabilities are inherited from its underlying inference engine (typically Ollama). A recent practical evaluation by Docker revealed significant challenges with local model tool calling, including eager invocation (models calling tools unnecessarily), incorrect tool selection, and difficulties handling tool responses properly. While Docker Model Runner supports tool calling through its OpenAI-compatible API when using appropriate models, the reliability varies greatly depending on the specific model and configuration. The containerizatio [source]
With the addition of vLLM, Docker Model Runner is now compatible with the two most dominant open-source model formats: Safetensors and GGUF. While Model Runner abstracts the complexity of setting up the engines, understanding the difference between these formats helps in choosing the right tool for your infrastructure. [source]
Docker announced the vLLM integration on 2025-11-20. At launch it supported only x86_64 with NVIDIA GPUs, had slower startup than llama.cpp, and few vLLM-format models on Docker Hub. Docker recommends prototyping on llama.cpp and scaling to vLLM. [source]
A single-hardware benchmark (M1 Pro, 50 runs each, model unnamed, March 2025) found DMR 24.53 vs Ollama 23.65 mean tokens/sec, a 0-12% per-prompt difference; the author calls them comparable. Variance was high and the setup was one machine. [source]
Docker Model Runner vs Ollama: Direct Comparison [source]
If you’re specifically comparing Ollama with Docker’s native container approach, see our detailed breakdown of Docker Model Runner vs Ollama. That guide focuses on Docker integration, GPU configuration, performance trade-offs, and production deployment differences. [source]
When to Choose: Ideal for users who already use Docker extensively in workflows, need seamless container orchestration, value Docker’s ecosystem and tooling, and want simplified deployment pipelines. For a detailed analysis of the differences, see Docker Model Runner vs Ollama comparison which explores when to choose each solution for your specific use case. [source]
Safetensors (vLLM) vs. GGUF (llama.cpp): Choosing the Right Format [source]
Until now, developers had to choose between simplicity and performance. You could either run models easily (using simplified portable tools like Docker Model Runner with llama.cpp) or achieve maximum throughput (with frameworks like vLLM). [source]
Choose Docker Model Runner (DMR) if your team already relies heavily on Docker, OCI artifacts, and container orchestration. [source]
Key Insight: DMR brings Docker workflows to LLM deployment, while Ollama remains framework-agnostic with simpler standalone operation. Your existing infrastructure matters more than technical differences. [source]
Bottom Line: If you’re already using Docker extensively, DMR is the natural choice. If you want the absolute simplest experience regardless of infrastructure, choose Ollama. To compare these local options with cloud APIs and other self-hosted setups, check our LLM Hosting: Local, Self-Hosted & Cloud Infrastructure Compared guide. [source]
Comparing LLM Runtimes for Alfresco in Spring AI: Ollama vs Docker Runner [source]
Speedup Factors (Runner vs Ollama - based on tokens/sec): Prompt Prompt 1 1.12 Prompt 2 1.01 Prompt 3 1.06 Prompt 4 1.01 Prompt 5 1.00 [source]
Docker Model Runner: Native GPU support without complex nvidia-docker configuration. Automatically detects and uses available GPUs, significantly simplifying the Docker GPU experience compared to traditional containers. [source]
If you’re using NVIDIA GPUs and want to configure CUDA acceleration properly, see our detailed guide on adding NVIDIA GPU support to Docker Model Runner. It covers Docker daemon configuration, NVIDIA Container Toolkit setup, and how to verify that your LLM is actually using GPU memory instead of falling back to CPU inference. [source]

Facts and statements

Models are pulled from Docker Hub, any OCI-compliant registry, or Hugging Face; GGUF and Safetensors files can be packaged as OCI artifacts and published to any registry. [source]
Docker Model Runner intelligently routes your request: if you pull a GGUF model, it utilizes llama.cpp; if you pull a safetensors model, it leverages the power of vLLM. With Docker Model Runner, both can be pushed and pulled as OCI images to any OCI registry. [source]
DMR supports three inference engines: llama.cpp (default, GGUF, all platforms), vLLM (Safetensors, NVIDIA GPUs; Linux x86_64 and Windows WSL2) and Diffusers (image generation, NVIDIA on Linux). [source]
Engine routing is automatic by model format: a GGUF model uses llama.cpp, a Safetensors model uses vLLM. The vLLM backend is installed once with 'docker model install-runner --backend vllm --gpu cuda'. [source]
| Engine | Best for | Model format | |--------|----------|--------------| | [llama.cpp](/ai/model-runner/inference-engines/#llamacpp) | Local development, resource efficiency | GGUF (quantized) | | [vLLM](/ai/model-runner/inference-engines/#vllm) | Production, high throughput | Safetensors | | [Diffusers](/ai/model-runner/inference-engines/#diffusers) | Image generation (Stable Diffusion) | Safetensors | [source]
DMR partners with Google, Hugging Face, and VMware Tanzu to expand the AI model ecosystem available through Docker Hub. If you’re new to Docker or need a refresher on Docker commands, our Docker Cheatsheet provides a comprehensive guide to essential Docker operations. [source]
Docker Model Runner: Leverages Docker’s security model with container isolation. Models packaged as OCI Artifacts can be scanned and signed. Distribution through Docker Hub enables access control and vulnerability scanning for enterprise users. [source]
Docker Model Runner Container workflows ⭐⭐⭐ Alpha/Beta ⚠️ Limited Docker Desktop GGUF (depends) NVIDIA, AMD Partial [source]
Today, we’re excited to announce that Docker Model Runner now integrates the vLLM inference engine and safetensors models, unlocking high-throughput AI inference with the same Docker tooling you already use. [source]
GGUF (GPT-Generated Unified Format): The native format for llama.cpp, GGUF is designed for high portability and quantization. It is excellent for running models on commodity hardware where memory bandwidth is limited. It packages the model architecture and weights into a single file. [source]
Docker Model Runner: Launched April 2025 with strong enterprise backing. Partnerships with Google, Hugging Face, and VMware Tanzu AI Solutions ensure broad model availability. Integration with Docker’s massive developer community (millions of users) provides instant ecosystem access. Still building community-specific resources as a new product. [source]
Docker Model Runner — a CLI-and-API feature bundled with Docker Desktop/Engine for developers already working in a Docker-based workflow, another option to weigh against Foundry Local's native-SDK approach if your stack is container-centric. See the full Docker Model Runner review. [source]
Model flexibility: Works seamlessly with popular open-weight models like GPT-OSS, Qwen3, Mistral, Llama 3, and others in the safetensors format. [source]
Docker Model Runner uses native docker model pull and docker model run commands, packaging models as OCI artifacts. [source]
For production and enterprise: Both DMR and third-party solutions (vLLM, TGI) in Docker offer orchestration, monitoring, and scalability [source]
- On Linux, Docker Model Runner and its inference engines, such as Diffusers, run inside a container, which provides the isolation boundary. - On macOS and Windows, the engines don't run inside a container, so Docker Model Runner runs them in a sandboxed environment (seatbelt/sandbox-exec and Job Objects respectively) [source]
Docker Model Runner Integrates vLLM for High-Throughput Inference [source]
By bringing vLLM to Docker Model Runner, we’re bridging the gap between fast local experimentation and robust production inference. [source]
Running vLLM models with Docker Model Runner is as simple as installing the backend and running your model, no special setup required. [source]
This flexibility makes Docker Model Runner a first in the industry — no other tool lets you switch between multiple inference engines within a single, portable, containerized workflow. [source]
Command Style docker model pull/run/package ollama pull/run/list [source]
Convert model references: ollama run llama2 → docker model pull ai/llama2 and docker model run ai/llama2 [source]
The introduction of DMR narrows the gap between Docker and Ollama in terms of ease of use. Ollama still wins on simplicity for quick prototyping, while DMR excels for teams already invested in Docker workflows. Both approaches are actively developed, production-ready, and the ecosystem is mature enough that switching between them is relatively painless. [source]
Safetensors: The native format for vLLM and the modern standard for high-end inference, safetensors is built for high-throughput performance. [source]
docker model run ai/smollm2 "Explain Docker Model Runner" [source]
For a full reference of docker model commands, packaging options, configuration flags, and practical examples, see our detailed Docker Model Runner Cheatsheet: Commands & Examples. It covers model pulling, packaging, configuration, and best practices for deploying LLMs locally with Docker. [source]
The landscape of local LLM deployment has evolved significantly with Docker’s introduction of Docker Model Runner (DMR) in 2025. The choice now depends on your specific requirements: [source]
Docker Model Runner Cheatsheet: Commands & Examples [source]
Whether you're building generative AI applications, experimenting with machine learning workflows, or integrating AI into your software development lifecycle, Docker Model Runner provides a consistent, secure, and efficient way to work with AI models locally. [source]
Docker Model Runner is Docker’s relatively new entry into local LLM deployment, leveraging Docker’s containerization strengths with native integration, Docker Compose support for easy multi-container deployments, simplified volume management for model storage and caching, and container-native service discovery. [source]
We’re incredibly excited about this new chapter for Docker Model Runner, and we can’t wait to see what we can build together. Let’s get to work! [source]

Related concepts

Docker Model Runner — is a hyponym of Docker Model Runner relative to Ollama
inference engine — is a part of Docker Model Runner relative to Ollama
Ollama — is a related of Docker Model Runner relative to Ollama
Llama — is a instance of Docker Model Runner relative to Ollama
GGUF — is a part of Docker Model Runner relative to Ollama
Safetensors — is a part of Docker Model Runner relative to Ollama
OCI registry — is a part of Docker Model Runner relative to Ollama
Docker Hub — is a part of Docker Model Runner relative to Ollama
Hugging Face Hub — is a hyponym of Docker Model Runner relative to Ollama
OpenAI-compatible — is a part of Docker Model Runner relative to Ollama
Foundry Local — is a hyponym of Docker Model Runner relative to Ollama
LocalAI — is a hyponym of Docker Model Runner relative to Ollama
open-weight — is a synonym of Docker Model Runner relative to Ollama
quantization — is a part of Docker Model Runner relative to Ollama
general availability — is a problem of Docker Model Runner relative to Ollama

Children

No children recorded.

← the whole tree · 3D view· how to read this page