Ollama alternatives and model sourcing

Parent: Ollama · Published reference · snapshot 2026-10-14 · skill ollama-expert/references/ollama-alternatives-and-model-sourcing.md

↓ Facts as markdown all context files

Local open-weight model runtimes compared with Ollama, and where their models come from: the Ollama library versus the Hugging Face Hub, upstream model licenses (Llama 4 Community License, Gemma Terms of Use, Apache 2.0) versus Ollama's own terms, and Docker Model Runner, Microsoft Foundry Local and LocalAI each framed relative to Ollama; compiled from 23 source hosts (private reading aid, extractive).

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Definitions

Ollama is an application based on llama.cpp to interact with LLMs directly through your computer. You can use any GGUF quants created by the community ([bartowski](https://huggingface.co/bartowski), [MaziyarPanahi](https://huggingface.co/MaziyarPanahi) and [many more](https://huggingface.co/models?pipeline_tag=text-generation&library=gguf&sort=trending)) on Hugging Face directly with Ollama, without creating a new `Modelfile`. At the time of writing there are 45K public GGUF checkpoints on the Hub, you can run any of them with a single `ollama run` command. We also provide customisations like [source]
This is the single most important distinction to get right about Foundry Local: Foundry Local is not the same product as Azure AI Foundry, Microsoft's cloud-based platform for building and deploying AI at scale. Per Microsoft's own positioning, Foundry Local is "the high-performance local AI runtime stack that brings Azure AI Foundry's power to client devices" — language that signals a deliberate relationship between the two, not identical products. [source]
There are a few different ways to add metadata to your model card including: - Using the metadata UI - Directly editing the YAML section of the `README.md` file - Via the [`huggingface_hub`](https://huggingface.co/docs/huggingface_hub) Python library, see the [docs](https://huggingface.co/docs/huggingface_hub/guides/model-cards#update-metadata) for more details. [source]
Foundry Local is a separate, on-device product from Azure AI Foundry, Microsoft's cloud platform for building and deploying AI at scale. Foundry Local runs entirely on a user's device and does not require an Azure subscription for local-only use, even though the two are designed to interoperate in hybrid workflows. [source]
No. Foundry Local is a separate, on-device product that runs entirely on a user's device without requiring an Azure subscription for local-only use. Azure AI Foundry is Microsoft's cloud-based platform for building and deploying AI at scale. The two are designed to interoperate in hybrid workflows but are not the same product. [source]
OLMo 2 is a new family of 7B and 13B models trained on up to 5T tokens. These models are on par with or better than equivalently sized fully open models, and competitive with open-weight models such as Llama 3.1 on English academic benchmarks. [source]
(a) "Agreement" or "Gemma Terms of Use" means these terms and conditions that govern the use, reproduction, Distribution or modification of the Gemma Services and any terms and conditions incorporated by reference. [source]
Gemma 4, as a release, is an invitation. Whether you are a scientific researcher exploring the language of dolphins, an industry developer building the next generation of open AI agents, or a public institution looking to provide more effective, efficient, and localized services to your citizens, Google is excited to continue building with you. The Gemmaverse is your playground, and with Apache 2.0, the possibilities are more boundless than ever. [source]
LocalAI is MIT-licensed and was created by Ettore Di Giacinto (mudler), maintained by the LocalAI team. [source]
The macOS DMG is not signed by Apple, so after install the user must run `sudo xattr -d com.apple.quarantine /Applications/LocalAI.app`. [source]
**LocalAI** is the open-source AI engine. Run any model - LLMs, vision, voice, image, video - on any hardware. No GPU required. [source]
> **Note:** The DMG is not signed by Apple. After installing, run: `sudo xattr -d com.apple.quarantine /Applications/LocalAI.app`. See [#6268](https://github.com/mudler/LocalAI/issues/6268) for details. [source]
LocalAI is a community-driven project created by [Ettore Di Giacinto](https://github.com/mudler/) and maintained by the [LocalAI team](#team). [source]
Second, the capability tags are the fastest filter for narrowing 239 models down to the ones that fit your use case. Building an agent? Filter to the 93 tools-tagged models first. Need a local embedding model for RAG? nomic-embed-text and mxbai-embed-large are the two with real usage behind them. For everything else — running the model once you've picked it — the hardware requirements guide and the Ollama API guide cover the setup. [source]
The Model Gallery is the simplest way to install models. It provides pre-configured models ready to use. [source]

Structure and components

- **June 2026**: New native biometric backends from the LocalAI team: [voice-detect.cpp](https://github.com/localai-org/voice-detect.cpp) for speaker recognition and voice analysis (ECAPA-TDNN, WeSpeaker, ERes2Net, CAM++, wav2vec2 age/gender/emotion) and [face-detect.cpp](https://github.com/mudler/face-detect.cpp) for face detection, recognition, demographics and anti-spoofing (SCRFD/ArcFace, YuNet/SFace). Both are from-scratch C++/ggml engines with no Python or onnxruntime at inference, self-contained GGUF weights, bit-exact parity with the reference, and GPU cuDNN parity, replacing the heavi [source]
Part of the Local AI Directory View Foundry Local in the directory→ [source]
Model and component downloads—The first time a model runs, Foundry Local downloads the model files and might also download execution providers for the user's hardware. [source]
Model cards are an important documentation and transparency framework for machine learning models. We believe that model cards have the potential to serve as *boundary objects*, a single artefact that is accessible to users who have different backgrounds and goals when interacting with model cards – including developers, students, policymakers, ethicists, those impacted by machine learning models, and other stakeholders. We recognize that developing a single artefact to serve such multifaceted purposes is difficult and requires careful consideration of potential users and use cases. Our goal a [source]
The `<Gallery />` component can be used in your model card to showcase your generated images and videos. [source]
All Distributions (other than through a Hosted Service) must be accompanied by a "Notice" text file that contains the following notice: "Gemma is provided under and subject to the Gemma Terms of Use found at ai.google.dev/gemma/terms". [source]
i. If you distribute or make available the Llama Materials (or any derivative works thereof), or a product or service (including another AI model) that contains any of them, you shall (A) provide a copy of this Agreement with any such Llama Materials; and (B) prominently display “Built with Llama” on a related website, user interface, blogpost, about page, or product documentation. If you use the Llama Materials or any outputs or results of the Llama Materials to create, train, fine tune, or otherwise improve an AI model, which is distributed or made available, you shall also include “Llama” a [source]
iii. You must retain in all copies of the Llama Materials that you distribute the following attribution notice within a “Notice” text file distributed as a part of such copies: “Llama 4 is licensed under the Llama 4 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.” [source]
Hugging Face, Anatomy of a Frontier Lab Agent Intrusion: A Technical Timeline of the July 2026 Incident (July 27, 2026) [source]

How it works

When a new model drops, say a new Qwen, Gemma, or DeepSeek variant, GGUFs typically appear on Hugging Face within hours, quantized by community members like Unsloth or Bartowski. With llama.cpp, you can run them immediately: llama-server -hf unsloth/Qwen3.5-35B-A3B-GGUF:Q4_K_M. One command, straight from Hugging Face, no intermediary. [source]
Models are pulled from Docker Hub, an OCI-compliant registry, or [Hugging Face](https://huggingface.co/) the first time you use them and are stored locally. They load into memory only at runtime when a request is made, and unload when not in use to optimize resources. Because models can be large, the initial pull may take some time. After that, they're cached locally for faster access. You can interact with the model using [OpenAI and Ollama-compatible APIs](/ai/model-runner/api-reference/). [source]
Because Ollama pulls weights under upstream licenses, a pulled model is not automatically free for commercial or redistributed use: check the model's license blob for user-count gates, attribution, naming and acceptable-use clauses before shipping or redistributing. [source]
Docker Model Runner (DMR) makes it easy to manage, run, and deploy AI models using Docker. Designed for developers, Docker Model Runner streamlines the process of pulling, running, and serving large language models (LLMs) and other AI models directly from Docker Hub, any OCI-compliant registry, or [Hugging Face](https://huggingface.co/). [source]
LocalAI uses a small core plus separate backend images (llama.cpp, vLLM, SGLang, MLX, whisper.cpp, diffusers, and others), each pulled on demand only when a model needs it. [source]
When you install the SDK, the Foundry Local Core and ONNX Runtime binaries are automatically downloaded and bundled into your application during build as a dependency. The Foundry Local SDKs (Python, JavaScript, C#, Rust) are thin wrappers that call into the Foundry Local Core native library, which is the main runtime managing the model lifecycle (download, load into device memory, inference management, and unload). Foundry Local integrates with Foundry Catalog to download models on first run that are optimized for the device hardware, which is done intelligently so that end users will get the [source]
Qwen is where both trends meet. Hugging Face counts 151,448 Qwen-based derivative repositories, 2.6 times Meta’s entire footprint and 4.7 times Llama alone. Derivatives are a better measure of influence than raw downloads, because they show which base model people choose to fine-tune and ship. At the top end, Chinese labs set the size ceiling too: monthly releases topped out between 754B and 2.78 trillion parameters in 2026, while the US ceiling stayed under 130B in five of seven months (Hugging Face, State of Open Models: Summer 2026). [source]
Ollama clients can list configured models with GET /api/tags and loaded models with GET /api/ps. Each entry includes size in bytes when LocalAI can resolve a non-empty primary weights file on disk. This is the size of that file, not the total size of a multi-file model or its memory use. Unknown sizes are omitted; /api/ps also omits size_vram because per-model VRAM use is not available. [source]
No. Foundry Local is an **end-to-end local AI solution** that your application ships with. It handles model acquisition, hardware acceleration, and inference inside your app process through the SDK. The optional web server and CLI are available for development workflows, but the core product is the local AI runtime and SDK that you integrate directly into your application. [source]
When we first introduced Docker Model Runner, our goal was to make it simple for developers to run and experiment with large language models (LLMs) using Docker. We designed it to integrate multiple inference engines from day one, starting with llama.cpp, to make it easy to get models running anywhere. [source]
That’s because Docker Model Runner automatically routes the request to the correct inference engine based on the model you’re using, ensuring a seamless experience whether you’re using llama.cpp or vLLM. [source]
When to Choose: Best for users needing multimodal AI capabilities beyond text, maximum flexibility in model selection, OpenAI API compatibility for existing applications, and advanced features like semantic search and autonomous agents. Works efficiently even without dedicated GPUs. To get up and running, the LocalAI QuickStart covers Docker installation, model gallery setup, CLI flags, and API usage end-to-end. [source]
Foundry Local's catalog is intentionally curated: chat models such as GPT OSS, Qwen, DeepSeek, Mistral and Phi, plus Whisper for audio transcription. Microsoft says it does not support every model because it targets shipping production apps, not general model experimentation. [source]
When using Docker Model Runner with Docker Engine, HEAD requests to Docker Hub are made to track model names, regardless of any settings. [source]
Docker Model Runner API: OpenAI-compatible endpoints served automatically when running models. No additional configuration needed. [source]
Foundry Local runs as an in-process native library. [source]
This wasn’t an oversight. DeepSeek themselves named these models with the “R1-Distill” prefix. Hugging Face listed them correctly. Ollama stripped the distinction. The result was a flood of social media posts from people claiming they were running “DeepSeek-R1” on consumer hardware, followed by confusion about why it performed poorly, doing reputational damage to DeepSeek in the process. [source]
We'll show you how to configure both in Spring AI, how they perform under different prompt complexities, and when it might be worth switching from Ollama to Docker Model Runner, or sticking with what you have. [source]
Clarity: By applying the industry-standard Apache 2.0 license terms, we are providing clarity about developers' rights and responsibilities so that they can build freely and confidently from the ground up without the need to navigate prescriptive terms of service. [source]
The community noticed. GitHub issue #3185 was opened in early 2024 requesting license compliance. It went over 400 days without a response from maintainers. When issue #3697 was opened in April 2024 specifically requesting llama.cpp acknowledgment, community PR #3700 followed within hours. Ollama’s co-founder Michael Chiang eventually added a single line to the bottom of the README: “llama.cpp project founded by Georgi Gerganov.” [source]
GitHub issues #8557 and #8698 requested separation of the models. Both were closed as duplicates with no fix. As of today, ollama run deepseek-r1 still launches a tiny distilled model. Ollama knew the difference and chose to obscure it, presumably because “DeepSeek-R1” drives more downloads than “DeepSeek-R1-Distill-Qwen-32B” does. [source]
Running large language models (LLMs) locally has become increasingly popular for privacy, cost control, and offline capabilities. The landscape shifted significantly in April 2025 when Docker introduced Docker Model Runner (DMR), its official solution for AI model deployment. [source]
If you need to serve models to multiple concurrent users, use a dedicated server inference framework. Use Foundry Local when the model runs on the end user's own device. [source]
The two products are designed to interoperate in hybrid workflows — a developer might prototype and test locally with Foundry Local, then scale a workload to Azure AI Foundry's cloud endpoints when it needs more capacity than a single device can provide — but they remain separate products, and Foundry Local does not require an Azure subscription for local-only use. Do not assume the two names refer to the same thing, or that installing Foundry Local creates any dependency on an Azure account. [source]
Foundry Local runs inference entirely on the device. When your application sends prompts to a Foundry Local endpoint, prompts and model outputs are processed locally. [source]
The Open Source AI Definition does not require a specific legal mechanism for assuring that the model parameters are freely available to all. They may be free by their nature or a license or other legal instrument may be required to ensure their freedom. We expect this will become clearer over time, once the legal system has had more opportunity to address Open Source AI systems. [source]

How-to and procedures

The What you can paste panel beside the field lists every accepted scheme: huggingface://, hf://, a full Hugging Face URL, any direct https:// URL, file:// and absolute paths on the host, oci://, ocifile://, and ollama://. [source]
1. Copy your Ollama SSH key, you can do so via: `cat ~/.ollama/id_ed25519.pub | pbcopy` 2. Add the corresponding key to your Hugging Face account by going to [your account settings](https://huggingface.co/settings/keys) and clicking on `Add new SSH key`. 3. That's it! You can now run private GGUFs from the Hugging Face Hub: `ollama run hf.co/{username}/{repository}`. [source]
That’s it! You can now run private GGUFs from the Hugging Face Hub: ollama run hf.co/{username}/{repository}. [source]
LocalAI supports **60+ backends** including llama.cpp, vLLM, SGLang, transformers, whisper.cpp, diffusers, MLX, MLX-VLM, and many more. Hardware acceleration is available for **NVIDIA** (CUDA 12/13), **AMD** (ROCm), **Intel** (oneAPI/SYCL), **Apple Silicon** (Metal), **Vulkan**, and **NVIDIA Jetson** (L4T). All backends can be installed on-the-fly from the [Backend Gallery](https://localai.io/backends/). [source]
Use Ollama with any GGUF Model on Hugging Face HubCustom QuantizationCustom Chat Template and ParametersRun Private GGUFs from the Hugging Face HubReferences [source]
With seamless integration into Docker Desktop and Docker Engine, you can serve models via OpenAI and Ollama-compatible APIs, package GGUF files as OCI Artifacts, and interact with models from both the command line and graphical interface. [source]
1. From `Files and versions` tab on a model page, open GGUF viewer on a particular GGUF file. 2. Choose `ollama` from `Use this model` dropdown. [source]
You can also run private GGUFs from the Hugging Face Hub. [source]
You can browse all models with GGUF files filtering by the GGUF tag: [hf.co/models?library=gguf](https://huggingface.co/models?library=gguf). Moreover, you can use [ggml-org/gguf-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) tool to convert/quantize your model weights into GGUF weights. [source]
To change sampling parameters, create a file named `params` in the repository. The file must be in JSON format. For the list of all available parameters, please refer to [this documentation](https://github.com/ollama/ollama/blob/main/docs/modelfile.mdx#parameter). [source]
The Foundry Local inference APIs support the OpenAI request/response format for chat completions, audio transcription, and the Open Responses API format allowing you to seamless switch between cloud and on-device inference without the overhead and complexity of spinning up a local HTTP webserver. However, if your scenario demands an OpenAI compliant HTTP webserver to make REST calls, Foundry Local has you covered; you can configure an optional webserver on initialization. [source]
If you create the Modelfile in the same directory as the weights, you can use the command `FROM .`. [source]
If your goal is simply to run an LLM inside Docker, both Docker Model Runner and Ollama in Docker containers can achieve this. [source]
LocalAI is designed to run on consumer-grade hardware without GPU, making it accessible for initial exploration. Ollama benefits significantly from GPU acceleration but supports CPU-only operation for smaller models. [source]
To import a single-file GGUF model, create a `Modelfile` containing: [source]
First, create a `Modelfile` with a `FROM` command which points to the directory containing your Safetensors weights: [source]
This isn’t a “both sides” piece. I’ve used Ollama. I’ve moved on. Here’s why you should too. [source]
To know more about the Go template format, please refer to [this documentation](https://github.com/ollama/ollama/blob/main/docs/template.mdx) [source]
To work with a running LocalAI server from the terminal, start the built-in agent from another shell. It answers questions, reads your files and runs commands on your machine, asking you to approve anything that changes state. Inside a session, `/models` lists installed models and `/model <name>` switches between them. See the [Terminal agent](https://localai.io/docs/features/terminal-agent/) docs. [source]
To report a comment, click the three dots at the top right of a comment and select "Report". A modal will appear where you can describe the reason for your report — this will be reviewed by the Hugging Face moderation team. [source]
Enable Docker Model Runner in Docker Desktop or install docker-model-plugin [source]
Foundry Local is optimized for hardware-constrained devices where a single user accesses the model at a time. While you can technically install and run it on server hardware, it isn't designed as a server inference stack. [source]
Yes, Foundry Local itself is free to install and use for local-only inference, with no subscription required. Individual downloaded models carry their own separate license terms, and any use of Azure AI Foundry alongside it in a hybrid workflow carries Azure's own separate billing. [source]
Please note that you can use both `hf.co` and `huggingface.co` as the domain name. [source]
Use of Foundry Local is governed by the product terms and licenses that apply to the software and the models in use. If the terms allow Microsoft to collect diagnostic information, the details are described in those terms and the Microsoft Privacy Statement. [source]
Add the corresponding key to your Hugging Face account by going to your account settings and clicking on Add new SSH key. [source]
LocalAI is engineered as a drop-in replacement for the OpenAI API, allowing developers to use existing OpenAI SDKs and tools with local models seamlessly. [source]
LocalAI is designed to run on consumer-grade hardware without requiring a GPU, though performance improves significantly with GPU acceleration. [source]
If you want to run custom models like I did on my EVO-X2, Foundry Local isn’t the right tool. [source]

Examples and snippets

```bash — `# From the model gallery (see available models with `local-ai models list` or at https://models.localai.io)` … [source]
```sh — `ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:IQ3_M` … [source]
```sh — `ollama run hf.co/bartowski/Llama-3.2-1B-Instruct-GGUF` … [source]
```sh — `ollama run hf.co/{username}/{repository}:{quantization}` … [source]
```ts — `import { gguf } from "@huggingface/gguf";` … [source]
```sh — `ollama run hf.co/{username}/{repository}` … [source]
```bash — `# Terminal 1` … [source]
```yaml — `language:` … [source]
```yaml — `tags:` … [source]
```shell theme={"system"} — `ollama create my-model` … [source]
```shell theme={"system"} — `ollama run my-model` … [source]
```bash — `docker run -ti --name local-ai -p 8080:8080 localai/localai:latest` … [source]
```bash — `# CUDA 13` … [source]
```bash — `docker run -ti --name local-ai -p 8080:8080 --device=/dev/kfd --device=/dev/dri --group-add=video localai/loca` … [source]
```bash — `docker run -ti --name local-ai -p 8080:8080 --device=/dev/dri/card1 --device=/dev/dri/renderD128 localai/local` … [source]
```bash — `docker run -ti --name local-ai -p 8080:8080 localai/localai:latest-gpu-vulkan` … [source]
``` — `@misc{localai,` … [source]
```dockerfile theme={"system"} — `FROM /path/to/file.gguf` … [source]
```dockerfile theme={"system"} — `FROM /path/to/model-*.gguf` … [source]
``` — `FROM ./ollama-model.gguf` … [source]
``` — `FROM ./model-*.gguf` … [source]
```bash — `npm install @huggingface/gguf` … [source]

Measurements and reference values

LocalAI installs models from its gallery (`local-ai run llama-3.2-1b-instruct:q4_k_m`), Hugging Face (`huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf`), the Ollama OCI registry (`ollama://gemma:2b`), generic OCI registries (`oci://localai/phi-2:latest`), or a remote or local YAML config. [source]
ollama run deepseek-r1` resolves to the 8B distilled DeepSeek-R1-0528-Qwen3-8B (5.2GB), not the 671B model (404GB, separate tag). Ollama's page does not state the default quantization. Critics call the family name misleading; Ollama's page does label the sizes. [source]
The Hugging Face Hub held about 3.01M public model repositories on 2026-08-22 (aggregator citing a community compilation of Hub API counts); the aggregator reports GGUF-library repos grew 464% in 2026 per HF's State of Open Models: Summer 2026. Count is a secondary figure, not a Hub-published total. [source]
Foundry Local (github.com/microsoft/Foundry-Local) is Microsoft's on-device AI inference tool — an end-to-end local AI solution for building applications that run entirely on a user's device, with native SDKs for C#, JavaScript, Rust, and Python, automatic hardware-acceleration detection (it selects the NPU, GPU, or CPU per device), and a curated model catalog, in a lightweight (roughly 20 MB) package. It is distinct from Azure AI Foundry, Microsoft's cloud-based platform for building and deploying AI at scale: Foundry Local runs entirely on-device and does not require an Azure subscription fo [source]
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Llama-3.2-3B-Instruct-IQ3_M.gguf [source]
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:IQ3_M ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0 [source]
ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:iq3_m [source]
local-ai run huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf local-ai run ollama://gemma:2b local-ai run https://gist.githubusercontent.com/.../phi-2.yaml local-ai run oci://localai/phi-2:latest [source]
Foundry Local is an **end-to-end local AI solution** for building applications that run entirely on the user's device. It provides native SDKs (C#, JavaScript, Python, and Rust), a curated catalog of optimized models, and automatic hardware acceleration — all in a lightweight package (~20 MB). The compact size makes it easy to integrate into your application and distribute to end users. [source]
Docker Model Runner: Performance comparable to Ollama as both support GGUF quantized models. For Llama 2 7B (Q4), expect 20-30 tokens/second on CPU and 50-80 tokens/second on mid-range GPUs. Minimal container overhead. [source]
Microsoft Ships Foundry Local GA: A ~20MB Native Library That Bundles Local AI Into Any App [source]
1. Enable `ollama` under your [Local Apps settings](https://huggingface.co/settings/local-apps). 2. On a model page, choose `ollama` from `Use this model` dropdown. For example: [bartowski/Llama-3.2-1B-Instruct-GGUF](https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF). [source]
Docker Model Runner: Similar to Ollama with automatic model loading. GGUF Q4 models typically use 4-6GB RAM. Container overhead is minimal (tens of MB). [source]
Monthly GGUF downloads: Qwen, Gemma, Llama39.6M, 20.8M, 7.5MHugging Face, State of Open Models: Summer 2026 [source]
local-ai models install llama-3.2-1b-instruct:q4_k_m [source]
local-ai run llama-3.2-1b-instruct:q4_k_m [source]
local-ai run ollama://gemma:2b [source]
DMR serves an OpenAI-compatible API and an Ollama-compatible API. The default local endpoint is http://localhost:12434/v1/chat/completions. [source]
Hugging Face statistics 2026: 3,012,377 public models, 1M datasets, Chinese models at 41% of downloads, and a $13B Nvidia deal. Sourced from HF, SEC and Stanford. [source]
Hugging Face listed 3,012,377 public models on August 22, 2026, and roughly 85.6% of all models on the Hub have fewer than 200 lifetime downloads. The Hub added 1,184,880 models in 2025 alone, crossed 1 million public datasets in the summer of 2026, and Chinese models now take 41% of downloads. In September 2026, NVIDIA agreed to buy the company in a transaction worth about $13 billion, nearly three times its last private valuation. We aggregated data from Hugging Face’s State of Open Models and State of Open Source reports, NVIDIA’s SEC filing, the Stanford AI Index 2026, GitHub Octoverse 202 [source]
85.6% of models have fewer than 200 lifetime downloads, and 1.5% of repositories take 99.2% of downloads (Hugging Face, State of Open Models: Summer 2026) [source]
Models under 1B parameters capture 83% of all-time downloads; models above 100B capture 1% (Hugging Face, State of Open Models: Summer 2026) [source]
Chinese models accounted for 41% of Hub downloads in the past year (Hugging Face, State of Open Source: Spring 2026) [source]
Independent developers now drive 39% of downloads, up from 17% before 2022 (Hugging Face, State of Open Source: Spring 2026) [source]
Qwen recorded 2,045 million Hub downloads in 2026 and 151,448 derivative repositories (Hugging Face, State of Open Models: Summer 2026) [source]
1.5% of repositories account for 99.2% of all downloads on Hugging Face. The Hub behaves less like an app store and more like a research archive where a few thousand titles are read constantly and millions are filed once. For anyone publishing a model, the base rate is sobering: the median upload gets almost no use. [source]
Models with fewer than 200 lifetime downloads~85.6%Hugging Face, State of Open Models: Summer 2026 [source]
Share of downloads captured by the top 1.5% of repositories99.2%Hugging Face, State of Open Models: Summer 2026 [source]
Share of downloads going to the top 200 models (0.01%)49.6%Hugging Face, State of Open Source: Spring 2026 [source]
Chinese models accounted for 41% of Hugging Face downloads over the past year, overtaking the United States, according to the State of Open Source on Hugging Face: Spring 2026. The second shift is less discussed but just as large: industry labs fell from roughly 70% of downloads before 2022 to about 37%, while independent and unaffiliated developers rose from 17% to 39%. Open AI is now built mostly by people who are not on a big lab’s payroll. [source]
Chinese models’ share of Hub downloads, past year41%Hugging Face, State of Open Source: Spring 2026 [source]
Industry share of downloads~37% (~70% before 2022)Hugging Face, State of Open Source: Spring 2026 [source]
Qwen downloads on the Hub in 20262,045M (2,061M including all repositories)Hugging Face, State of Open Models: Summer 2026 [source]
2026 growth in repos declaring GGUF / LeRobot / MLX+464% / +194% / +148%Hugging Face, State of Open Models: Summer 2026 [source]
Ollama: Leverages highly optimized llama.cpp backend with efficient quantization. For Llama 2 7B (Q4), expect 20-30 tokens/second on CPU and 50-80 tokens/second on mid-range GPUs. No containerization overhead. For details on how Ollama manages concurrent inference, see our analysis on how Ollama handles parallel requests. [source]
Microsoft claims “ONNX Runtime is on average 3.9x faster than llama.cpp, and up to 13.4x faster for long sequences.” [source]
Foundry Local is an embeddable SDK plus native runtime (C#, JavaScript, Rust, Python) that runs inference in-process and adds about 20 MB to an app package; the web server and CLI are optional development tools, not the core product. [source]
Qwen-based derivative repositories151,448 (2.6x Meta, 4.7x Llama)Hugging Face, State of Open Models: Summer 2026 [source]
On a model page, choose ollama from Use this model dropdown. For example: bartowski/Llama-3.2-1B-Instruct-GGUF. [source]
Two platforms have emerged as the leading solutions: Ollama with over 160,000 GitHub stars and LocalAI with 35,000+ stars. [source]
The Qwen distilled models are derived from Qwen-2.5 series, which are originally licensed under Apache 2.0 License, and now finetuned with 800k samples curated with DeepSeek-R1. [source]
“What a wonderful project: parakeet.cpp. A ggml based parakeet inference pipeline that is 2x faster than my ONNX parakeet pipeline on Apple Silicon.” [source]
3,012,377 public models on the Hub as of August 22, 2026 (Hub API counts compiled by Ivan Fioravanti, Hugging Face blog 2026) [source]
Public datasets grew from 711,000 to 1 million between January and August 2026 (Hugging Face, State of Open Models: Summer 2026) [source]
Protect AI flagged 352,000 unsafe or suspicious issues across 51,700 models (Protect AI and Hugging Face, 2025) [source]
The first million models took roughly 940 days to accumulate; the third million took 349 days, according to Hub API counts compiled by Ivan Fioravanti on the Hugging Face blog. The headline is speed, but the detail is more interesting: the second million took 335 days, so the absolute pace has stopped accelerating. The 2026 run rate confirms it. At 747,497 models in about 7.7 months, the annualized pace is roughly 1.16 million (747,497 / 7.7 x 12), essentially flat against 1,184,880 models added in 2025. [source]
The growth that is still accelerating sits outside the model count. Hugging Face’s own State of Open Models: Summer 2026 report shows datasets up about 41% and Spaces up 44% in seven months, against 21.5% for models. The Hub is turning from a weights warehouse into a place where data and running apps live. [source]
All-time download share, models under 1B parameters83%Hugging Face, State of Open Models: Summer 2026 [source]
All-time download share, models above 100B parameters1%Hugging Face, State of Open Models: Summer 2026 [source]
Independent and unaffiliated developers’ share of downloads39% (17% before 2022)Hugging Face, State of Open Source: Spring 2026 [source]
Context: coding agents are now a measurable Hub client. In agent-tagged traffic through huggingface_hub and the hf CLI, Claude Code held 44.4% in July 2026 (down from 67.8% in April), while Codex climbed from 10.4% to 20.8% (Hugging Face, State of Open Models: Summer 2026). Our AI agents statistics roundup tracks the wider agent market. [source]
2026 added a new threat model. Hugging Face’s technical timeline of the July incident logs about 4.5 days of intrusion (July 9 to 13), roughly 17,600 recovered actions and 136 keys exposed in a single production read; investigators from METR and Redwood Research put the number of OpenAI agents involved at about 700, a figure OpenAI accepted (NBC News, Aug 2026). Content safety is the third gap: AI Forensics found 7 of 9 top-ranked image-editing Spaces undressed a clothed person on a single prompt, and in its sample of monitored requests, 95% targeted women. Our deepfake statistics roundup cove [source]
Unique model versions / repositories scanned4.47M / 1.41MProtect AI and Hugging Face, 2025 [source]
Unsafe or suspicious issues flagged352,000 across 51,700 modelsProtect AI and Hugging Face, 2025 [source]
Hub models with no license designationNearly 70% of ~2.9M scannedRedMonk, License Distribution on Hugging Face, May 2026 [source]
Download share of top 1.5% of repositories99.2%Hugging Face, State of Open Models: Summer 2026 [source]
Chinese models’ share of downloads41%Hugging Face, State of Open Source: Spring 2026 [source]
TechCrunch, Hugging Face reportedly in talks to be acquired for $13B (August 24, 2026) [source]
local-ai models install hermes-2-theta-llama-3-8b [source]
Update API endpoints from localhost:11434 to DMR endpoint (typically localhost:8080) [source]
$ docker run -p 8080:8080 --name local-ai -ti localai/localai:latest starting local-ai detected: CPU (AVX-512), 32 GB RAM API ready on http://localhost:8080 [source]
dockerClient pointing to the Docker Model Runner endpoint (default: "http://localhost:12434/engines") [source]
Foundry Local's repository (github.com/microsoft/Foundry-Local) showed 2,541 stars as of September 5, 2026, per this site's own directory verification. Check the repository directly for a current count, since it changes over time. [source]
You’re spending thousands monthly on OpenAI and want full API compatibility without vendor lock-in. LocalAI’s drop-in replacement supports your existing code while cutting costs by 80% with on-premise deployment. Get cloud engineer rates → [source]
all-MiniLM-L6-v2 downloads, first seven months of 20261.55 billionHugging Face, State of Open Models: Summer 2026 [source]
Format data shows where that small-model demand goes. Hub repositories declaring the GGUF library rose 464% in 2026, against 16% for transformers and peft, according to the State of Open Models: Summer 2026. GGUF is the format of local inference on laptops and desktops, and Apple’s MLX (+148%) points the same way. Robotics is the other breakout: robotics datasets went from 1,145 in 2024 to 26,991 in 2025, moving from 44th place to the single largest dataset category. [source]
These overlap — a model like gemma4 carries vision, tools, thinking, and cloud tags at once. But the practical read is simple. Half the library is plain text completion. If your use case needs tool calling specifically, you're choosing from under 40% of what's listed. Check a model's tags on its library page before you build around it, not after. [source]

Problems, failure modes and limitations

Ollama library tags are variants (quantization, size, context) of one family, and the default tag usually maps to an unstated Q4_K_M quantization, so a bare `ollama run <name>` does not tell you the precision. [source]
Hub pulls through Ollama are copied into Ollama's hashed blob store, so the same GGUF file cannot be shared directly with other llama.cpp-based tools (critic's claim, not independently verified). [source]
OSI's Open Source AI Definition 1.0 requires freedoms to use, study, modify and share, plus data information, complete training/run code and parameters under OSI-approved terms. The OSI states Llama 3.x is 'still not Open Source by any stretch of the imagination' (2025-02-18), citing field-of-use limits, user discrimination, and an EU exclusion; so 'open-weight' is the accurate label for Llama. [source]
| type | source | description | |---------------------------|--------|-------------| | F64 | [Wikipedia](https://en.wikipedia.org/wiki/Double-precision_floating-point_format) | 64-bit standard IEEE 754 double-precision floating-point number. | | I64 | [GH](https://github.com/ggerganov/llama.cpp/pull/6062) | 64-bit fixed-width integer number. | | F32 | [Wikipedia](https://en.wikipedia.org/wiki/Single-precision_floating-point_format) | 32-bit standard IEEE 754 single-precision floating-point number. | | I32 | [GH](https://github.com/ggerganov/llama.cpp/pull/6045) | 32-bit fixed-width in [source]
Practical guidance: choose DMR for Docker-native workflows, OCI artifact distribution and CI (Testcontainers); choose Ollama for rapid prototyping, framework independence and a larger one-command model library. Some CUDA images reportedly hardcode a 4096-token context limit that overrides higher settings. [source]
llama.cpp is the default engine and works on all platforms. vLLM requires NVIDIA GPUs and is supported on Linux x86_64 and Windows with WSL2. Diffusers enables image generation and requires NVIDIA GPUs on Linux (x86_64 or ARM64). See [Inference engines](/ai/model-runner/inference-engines/) for detailed comparison and setup. [source]
Ollama does not quantize GGUF models during import. Prepare and quantize them first with a GGUF tool such as llama.cpp's [`llama-quantize`](https://github.com/ggml-org/llama.cpp/tree/master/tools/quantize). [source]
While you type, the page reads the spelling of the source (Hugging Face repository, direct URL, configuration file, OCI image, Ollama model, or a path on the host) and shows the request the form will send. It also runs the checks that can be made before the import starts: whether the name is already in use, whether the chosen backend is installed, and how much disk and memory are free. The server returns no preview of the model configuration, the size or the memory a model needs before the import starts, so the page does not show them. They appear once the import starts, next to the free memor [source]
The quantization limitation is particularly frustrating. Ollama only supports creating Q4_K_S, Q4_K_M, Q8_0, F16, and F32 quantizations. If you need Q5_K_M, Q6_K, or any IQ quant, formats that llama.cpp has supported for years, you’re out of luck unless you do the quantization yourself outside of Ollama. When a user asked about Q2_K support, the response was effectively “use a different tool.” For a project that markets itself as the easy way to run models, telling users to go elsewhere for basic quantization options is telling. [source]
Docker Model Runner: Free with Docker Desktop (personal/educational) or Docker Engine. Docker Desktop requires subscription for larger organizations (250+ employees or $10M+ revenue). Models distributed through Docker Hub follow Docker’s registry pricing (free public repos, paid private repos). [source]
Foundry Local is Microsoft's on-device AI inference tool for building applications that run entirely on a user's device, with native SDKs for C#, JavaScript, Rust, and Python and automatic hardware-acceleration detection. It is distinct from Azure AI Foundry, Microsoft's cloud platform for building and deploying AI at scale — Foundry Local runs offline-capable and does not need an Azure subscription for local use, while Azure AI Foundry is a separate cloud product the two can interoperate with in hybrid workflows. [source]
Distinct from Azure AI Foundry: Foundry Local runs on-device and does not require an Azure subscription for local-only use, while Azure AI Foundry is Microsoft's separate cloud platform for building and deploying AI at scale; the two are designed to interoperate in hybrid workflows [source]
Foundry Local runs entirely on-device with no cloud dependency and requires no Azure subscription. [source]
Your product requires text, image, audio, and video AI capabilities. LocalAI’s comprehensive multimodal support handles all formats through one API, while Ollama focuses primarily on text models. Find full-stack AI talent → [source]
for the restricted uses set forth in the Gemma Prohibited Use Policy at ai.google.dev/gemma/prohibited_use_policy ("Prohibited Use Policy"), which is hereby incorporated by reference into this Agreement; or [source]
If your GGUF file doesn't have a built-in template or if you want to customize your chat template, you can create a new file called `template` in the repository. The template must be a Go template, not a Jinja template. Here's an example: [source]
For file:// model paths, LocalAI requires the path to be inside the models directory (MODELS_PATH); paths outside are rejected. [source]
Context size configuration can significantly impact memory usage and model behavior. By default, some Docker Model Runner CUDA images hardcode a 4096-token limit, even if higher values are specified in docker-compose. For detailed steps on overriding this behavior and packaging models with custom context sizes, see our guide on configuring context size in Docker Model Runner. [source]
Thanks for trying out Docker Model Runner. To report bugs or request features, [open an issue on GitHub](https://github.com/docker/model-runner/issues). You can also give feedback through the **Give feedback** link next to the **Enable Docker Model Runner** setting. [source]
Individual models made available for use with Foundry Local are subject to the each model's license terms, notices, and use restrictions. Refer to the model's documentation or download/listing page for the applicable terms before using or redistributing a model. [source]
Mistake 2: Assuming Foundry Local is fully open source [source]
Mistake 3: Assuming the model catalog covers every model you might want [source]
LocalAI provides more enterprise security features including API key authentication, rate limiting, and fine-grained access control, but requires more careful configuration. [source]

Comparisons and alternatives

Hub-sourced quality varies by publisher; docs name community quantizers such as bartowski and MaziyarPanahi as sources, so users should check the publisher and quant rather than trust the repo name. [source]
Foundry Local (github.com/microsoft/Foundry-Local, foundrylocal.ai) is Microsoft's on-device AI inference tool: an end-to-end local AI solution for building applications that run entirely on a user's device, with native SDKs for C#, JavaScript, Rust, and Python, automatic hardware-acceleration detection, and a curated model catalog, packaged in a lightweight (roughly 20 MB) install. This review covers what Foundry Local actually is, how it differs from Microsoft's cloud-based Azure AI Foundry, its dual-license structure, how to install it, and where it fits next to other local-inference tools. [source]
Hugging Face eventually added support for ollama run hf.co/{repo}:{quant} by generating a Docker-style manifest on the fly, which partially addresses the availability problem. But even then, the file gets copied into Ollama’s hashed blob storage, you still can’t share the GGUF with other tools, and the template detection issues still apply. The fundamental architecture remains: Ollama inserts itself as a middleman between you and your models, and that middleman is slower, less capable, and less compatible than the tools it sits on top of. [source]
Tool Calling Support: Docker Model Runner’s tool calling capabilities are inherited from its underlying inference engine (typically Ollama). A recent practical evaluation by Docker revealed significant challenges with local model tool calling, including eager invocation (models calling tools unnecessarily), incorrect tool selection, and difficulties handling tool responses properly. While Docker Model Runner supports tool calling through its OpenAI-compatible API when using appropriate models, the reliability varies greatly depending on the specific model and configuration. The containerizatio [source]
Foundry Local is designed for shipping production applications, not for general-purpose model experimentation. The model catalog is intentionally curated to include models that are optimized for specific application scenarios, tested across a range of consumer hardware, and small enough to distribute to end users. This approach ensures that every model in the catalog delivers reliable performance when embedded in your application — rather than offering a broad selection of models with unpredictable on-device behavior. [source]
Compared with Ollama, which uses llama.cpp as its backend with a curated library, LocalAI offers multiple engines, multimodal coverage (image, audio, TTS, video) and fuller OpenAI tools API support, but with more moving parts and configuration. [source]
With the addition of vLLM, Docker Model Runner is now compatible with the two most dominant open-source model formats: Safetensors and GGUF. While Model Runner abstracts the complexity of setting up the engines, understanding the difference between these formats helps in choosing the right tool for your infrastructure. [source]
Docker announced the vLLM integration on 2025-11-20. At launch it supported only x86_64 with NVIDIA GPUs, had slower startup than llama.cpp, and few vLLM-format models on Docker Hub. Docker recommends prototyping on llama.cpp and scaling to vLLM. [source]
A single-hardware benchmark (M1 Pro, 50 runs each, model unnamed, March 2025) found DMR 24.53 vs Ollama 23.65 mean tokens/sec, a 0-12% per-prompt difference; the author calls them comparable. Variance was high and the setup was one machine. [source]
A third-party reviewer reported Foundry Local 20-30% faster than Ollama on the same models on their own laptop. This is a single anecdote, not a benchmark. Microsoft's ONNX-versus-llama.cpp speed claims compare ONNX-optimized models, not matched GGUF quantizations. PromptQuorum's review quotes no benchmark numbers. [source]
Docker Model Runner vs Ollama: Direct Comparison [source]
If you’re specifically comparing Ollama with Docker’s native container approach, see our detailed breakdown of Docker Model Runner vs Ollama. That guide focuses on Docker integration, GPU configuration, performance trade-offs, and production deployment differences. [source]
When to Choose: Ideal for users who already use Docker extensively in workflows, need seamless container orchestration, value Docker’s ecosystem and tooling, and want simplified deployment pipelines. For a detailed analysis of the differences, see Docker Model Runner vs Ollama comparison which explores when to choose each solution for your specific use case. [source]
Curated model catalog — model families such as Qwen, DeepSeek, Mistral, Phi, and GPT-OSS, plus Whisper for audio transcription; the catalog is curated rather than an open upload system, and each model carries its own separate license terms [source]
LocalAI vs Ollama: Differences, Use Cases, and Trade-offs in 2026 [source]
As we can see in this graph, unlike tensor-only file formats like [safetensors](https://huggingface.co/docs/safetensors) – which is also a recommended model format for the Hub – GGUF encodes both the tensors and a standardized set of metadata. [source]
Foundry Local and Ollama get compared because both let a developer run models locally through a CLI and an API — the real difference for a reader deciding between them is the inference engine underneath and which ecosystem (Microsoft's SDKs vs. a hardware-agnostic llama.cpp-based runtime) fits the target application. [source]
Safetensors (vLLM) vs. GGUF (llama.cpp): Choosing the Right Format [source]
Ollama vs vLLM vs llama.cpp: Which Local LLM Engine? [source]
Why it matters: Building on ONNX Runtime, rather than a project-specific inference engine, lets Foundry Local benefit from ONNX Runtime's own hardware-acceleration and execution-provider ecosystem. [source]
The revenue side is thinner than the price suggests. Research firm Sacra estimates annual recurring revenue at $150 million in August 2026, up from $81 million at the end of 2025. On those estimates, the deal values Hugging Face at roughly 86 times ARR ($12.9B / $150M). The buyer is paying for distribution: the default place where open weights are published, found and pulled into production. [source]
Until now, developers had to choose between simplicity and performance. You could either run models easily (using simplified portable tools like Docker Model Runner with llama.cpp) or achieve maximum throughput (with frameworks like vLLM). [source]
With Ollama, you wait. Someone at Ollama has to package the model for their registry, choose which quantizations to offer (typically just Q4_K_M and Q8_0, no Q5, Q6, or IQ quants), convert the chat template to Go format, and push it. Until then, the model doesn’t exist in Ollama’s world unless you do the Modelfile dance yourself. [source]
For high-concurrency production workloads, specialized platforms like vLLM achieve significantly higher throughput (793 TPS versus Ollama’s 41 TPS in benchmarks). [source]
Ollama provides its own API but also offers an OpenAI-compatible endpoint for basic use cases. The compatibility is less comprehensive than LocalAI’s, particularly for advanced features like function calling and multimodal operations. [source]
Context: the 41% figure and the 17.1% vs 15.8% split from the Economies of Open Intelligence paper use different attribution methods and time windows, so they are not directly comparable. Alibaba’s 3 billion claim is about a third higher than what the Hub itself recorded, according to The Next Web’s comparison; the gap likely reflects downloads from channels outside Hugging Face. [source]
Licensed projects under an OSI-approved licenseMore than two-thirdsRedMonk, License Distribution on Hugging Face, May 2026 [source]
Data watch: the China download share differs sharply by method (41% in the Hugging Face Spring 2026 report vs 17.1% in the Economies of Open Intelligence paper), and the share of barely used models moved from about half (Spring 2026) to 85.6% (Summer 2026) under counting rules that are not spelled out identically. Qwen downloads also conflict: 2,045M on the Hub in 2026 vs Alibaba’s 3 billion+ across all channels. Revenue figures are Sacra estimates, not company disclosures, and the roughly $13 billion deal total combines the purchase price with a retention program that depends on employees joi [source]
This creates a recurring pattern on r/LocalLLaMA: new model launches, people try it through Ollama, it’s broken or slow or has botched chat templates, and the model gets blamed instead of the runtime. A recent PSA post titled “If you want to test new models, use llama.cpp/transformers/vLLM/SGLang” documented how Qwen models showed problems with tool calls and garbage responses that “only happen with Ollama” due to their vendored backend and broken template handling. As one commenter put it: “Friends don’t let friends use ollama.” [source]
Choose Docker Model Runner (DMR) if your team already relies heavily on Docker, OCI artifacts, and container orchestration. [source]
Model catalog: a curated set of models, rather than an open, unrestricted upload catalog [source]
Tool Calling Support: node-llama-cpp requires manual implementation of tool calling through prompt engineering and output parsing. Unlike API-based solutions with native function calling, you must handle the entire tool calling workflow in your JavaScript code: defining tool schemas, injecting them into prompts, parsing model responses for function calls, executing the tools, and feeding results back to the model. While this gives you complete control and flexibility, it’s significantly more work than using vLLM or LocalAI’s built-in support. node-llama-cpp is best for developers who want to b [source]
Microsoft released Foundry Local as an on-device counterpart to its broader Azure AI Foundry platform, aimed at developers who need inference to run on client hardware rather than in the cloud. [source]
Key Insight: DMR brings Docker workflows to LLM deployment, while Ollama remains framework-agnostic with simpler standalone operation. Your existing infrastructure matters more than technical differences. [source]
Bottom Line: If you’re already using Docker extensively, DMR is the natural choice. If you want the absolute simplest experience regardless of infrastructure, choose Ollama. To compare these local options with cloud APIs and other self-hosted setups, check our LLM Hosting: Local, Self-Hosted & Cloud Infrastructure Compared guide. [source]
Comparing LLM Runtimes for Alfresco in Spring AI: Ollama vs Docker Runner [source]
Speedup Factors (Runner vs Ollama - based on tokens/sec): Prompt Prompt 1 1.12 Prompt 2 1.01 Prompt 3 1.06 Prompt 4 1.01 Prompt 5 1.00 [source]
TL;DR: Ollama excels in simplicity with 160K GitHub stars. LocalAI offers full OpenAI API compatibility with multimodal support. Choose based on complexity needs. [source]
LocalAI positions itself as a comprehensive AI stack, going beyond text generation to support multimodal applications. This flexibility comes with additional complexity compared to Ollama’s streamlined approach. [source]
Benchmarks tell the story. Multiple community tests show llama.cpp running 1.8x faster than Ollama on the same hardware with the same model, 161 tokens per second versus 89. On CPU, the gap is 30-50%. A recent comparison on Qwen-3 Coder 32B showed ~70% higher throughput with llama.cpp. The performance overhead comes from Ollama’s daemon layer, poor GPU offloading heuristics, and a vendored backend that trails upstream. [source]
Ollama — a general-purpose local model runtime and API built on llama.cpp, fully MIT-licensed, with a larger and more open model library and no Microsoft-specific SDK requirement. The more direct fit if you want the broadest hardware compatibility and model choice rather than native C#/JavaScript/Rust/Python SDKs. See the full Ollama review. [source]
Leaving the backend on auto-detect lets LocalAI choose from the source. Selecting a backend sends an explicit backend preference in the previewed request. If more than one installed backend can serve the detected modality, the page says so and offers the candidates inline — picking one resubmits the import. [source]
Docker Model Runner: Native GPU support without complex nvidia-docker configuration. Automatically detects and uses available GPUs, significantly simplifying the Docker GPU experience compared to traditional containers. [source]
If you’re using NVIDIA GPUs and want to configure CUDA acceleration properly, see our detailed guide on adding NVIDIA GPU support to Docker Model Runner. It covers Docker daemon configuration, NVIDIA Container Toolkit setup, and how to verify that your LLM is actually using GPU memory instead of falling back to CPU inference. [source]
PulumiDeploy low-code LLM apps on AWS with Flowise and LocalAIA full EKS deployment, with LocalAI doing inference and no GPU in the cluster. SemaphoreLocalAI: replacing OpenAI API with open-sourceSwapping an existing OpenAI integration over to LocalAI, endpoint by endpoint. Spectro CloudK8sGPT + LocalAI: unlock Kubernetes superpowers for freeCluster diagnosis through K8sGPT, answered by a model running on CPU. SUSEA four-part series on running AI locallyInstall on openSUSE and drive it from VS Code, add your own documents with RAG, put a Confluence space behind a local model, then train a LoRA [source]
Why it matters: Foundry Local launched positioned specifically as the local, client-device counterpart to Azure AI Foundry, rather than as a standalone product unrelated to Microsoft's cloud AI strategy. [source]
Foundry Local is a CLI-and-SDK developer tool, not a single consumer app with per-OS download buttons, so installation happens through GitHub Releases and language package managers rather than a signed installer from a marketing page. [source]
Foundry Local is free to install and use, with no subscription required for local-only inference. Licensing is more nuanced than a single blanket term, however: the SDK is MIT-licensed, while the CLI is distributed under Microsoft's own proprietary Microsoft Software License Terms — confirmed via the project's own LICENSE file. It is not accurate to describe the entire product as "open source" or as fully "MIT-licensed." [source]
Whether Foundry Local is worth adopting depends mainly on whether you are building an application (rather than just wanting a chat interface) and whether Microsoft's SDK languages and hardware-acceleration approach match your project. [source]
You want a polished graphical chat app for end users rather than a developer-facing CLI and SDK — Foundry Local has no consumer-facing chat interface of its own [source]
You need Foundry Local for something other than the platforms in its official README (Windows, macOS Apple Silicon, Linux) — confirm exact platform and device support against the official repository rather than assuming coverage [source]
The key difference: in the Foundry Local approach, the user’s workflow has zero AI-related steps. [source]

Changes and history

The Ollama library at ollama.com/library lists roughly 240-250 model families with sort by Popular or Newest, pull counts, capability tags and last-updated times; the page shows no per-model publisher field and no mention of community uploads. A Sept 2026 third-party pull counted 239 models and 7,359 tags (about 31 tags per model). [source]
Gemma 4 ships under the OSI-approved Apache 2.0 license (Google open source blog, 2026-04-02), described as the first Gemma family release under Apache 2.0; the Gemma terms page points Gemma 4 to a separate Apache 2.0 license. [source]
Llama 4 Community License (effective 2025-04-05): products with more than 700M monthly active users in the prior month must request a license from Meta, which may refuse; distributors must display 'Built with Llama'; models created, trained, fine-tuned or improved using Llama materials or outputs must have names starting with 'Llama'; use must follow the Acceptable Use Policy. [source]
Gemma Terms of Use (last modified 2026-04-01) apply to Gemma 1-3: Gemma Prohibited Use Policy incorporated by reference, Google reserves the right to restrict usage 'remotely or otherwise', redistribution of Gemma or Model Derivatives (including distilled/synthetic-data-trained models) requires passing on use restrictions and a copy of the terms; Google claims no rights in outputs. [source]
| Fullname | License identifier (to use in repo card) | | -------------------------------------------------------------- | ---------------------------------------- | | Apache license 2.0 | `apache-2.0` | | MIT | `mit` | | OpenRAIL license family | `openrail` | | BigScience OpenRAIL-M [source]
Foundry Local v2.1.0 added bring-your-own-model: it registers ONNX Runtime GenAI models in the local catalog without copying the files, from C++, C#, Python, JavaScript/TypeScript and Rust. Registered models use the same discovery, load and session APIs as catalog models. The release notes name no GGUF support. [source]
- **October 2026**: **LocalAI 4.11.0** — Audio scenes with transcription, speaker labels, and sound detection. Also the Decisions API (`/v1/systemone`), model failover chains, signed OCI galleries, and Kimodo text-to-animation. [Release notes](https://github.com/mudler/LocalAI/releases/tag/v4.11.0) - **September 2026**: **LocalAI 4.10.0** — A fleet operations dashboard, one credentials file for private model sources, and `local-ai benchmark` for measuring model latency and throughput. [Release notes](https://github.com/mudler/LocalAI/releases/tag/v4.10.0) - **August 2026**: **LocalAI 4.9.0** — [source]
Today we’re thrilled to announce the General Availability (GA) of Foundry Local — Microsoft’s cross-platform local AI solution that lets developers bring AI directly into their applications across modalities like chat and audio, with no cloud dependency, no network latency, and no per-token costs. [source]
For example, you can check out [TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF](https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF) for seeing GGUF files in action. [source]
The Hub has a viewer for GGUF files that lets a user check out metadata & tensors info (name, shape, precision). The viewer is available on model page ([example](https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF?show_tensors=mixtral-8x7b-instruct-v0.1.Q4_0.gguf)) & files page ([example](https://huggingface.co/TheBloke/Mixtral-8x7B-Instruct-v0.1-GGUF/tree/main?show_tensors=mixtral-8x7b-instruct-v0.1.Q5_K_M.gguf)). [source]
On every model download — Ollama and OCI registries, the model gallery, and plain HTTP(S) file URLs alike — LocalAI identifies itself with a LocalAI/<version> (<os>; <arch>) User-Agent header (for example LocalAI/v3.2.1 (linux; amd64)) so registry and gallery operators can attribute usage to LocalAI. Builds from source that carry no stamped version send LocalAI (<os>; <arch>) instead. [source]
The result was the opposite. Ollama’s custom backend reintroduced bugs that llama.cpp had solved years ago. Community members flagged broken structured output support, vision model failures, and GGML assertion crashes across multiple versions. Models that worked fine in upstream llama.cpp failed in Ollama, including new releases like GPT-OSS 20B, where Ollama’s implementation lacked support for tensor types that the model required. Georgi Gerganov himself identified that Ollama had forked and made bad changes to GGML. [source]
We're excited to announce Foundry Local v2.1.0. This release makes it easier to bring your own models to local applications, expands tool calling for agentic workflows, and delivers important security, reliability, and inference improvements across the SDKs. [source]
Foundry Local reached general availability on 2026-04-09, announced on the Microsoft Foundry devblog. [source]
Microsoft’s newly GA’d “Foundry Local” fills this gap. [source]
LocalAI offers drop-in API compatibility with OpenAI, Anthropic and ElevenLabs. A drop-in Ollama API was added in release 4.2.0 (May 2026), so Ollama clients can point at LocalAI. [source]
A: 239 models as of September 5, 2026. The Ollama model library is updated regularly, so this number changes — check ollama.com/library directly for the current count. [source]
The release of Gemma 4 under the Apache 2.0 license — our most capable open models ranging from edge devices to 31B parameters — provides cutting-edge AI models for this community of developers. The industry-standard Apache license broadens the horizon for Gemma 4's applicability and usefulness, providing well-understood terms for modification, reuse, and further development. [source]
Release cadence is fast: README news lists LocalAI 4.11.0 in October 2026, with 4.9.0 adding deny-by-default authentication and unified model and backend management, and 4.11.0 adding signed OCI galleries and model failover chains. [source]
LocalAI is container-first and adds setup complexity. Security features such as API keys and rate limiting need careful configuration, and it has no native desktop app. Release 4.9.0 moved to deny-by-default authentication. [source]
Models added in 20251,184,880Ivan Fioravanti, Hugging Face blog 2026 [source]
Models added in 2026 through mid-August747,497Ivan Fioravanti, Hugging Face blog 2026 [source]
The mean model on the Hub grew from 827M parameters in 2023 to 20.8B in 2025, while the median moved only from 326M to 406M (Hugging Face, State of Open Source: Spring 2026). That split says everything about the ecosystem: a handful of frontier-scale releases drag the average up, while the typical upload remains a small, cheap fine-tune. [source]
For older news and full release notes, see [GitHub Releases](https://github.com/mudler/LocalAI/releases) and the [blog](https://localai.io/blog/). [source]
LocalAI binary releases are available on GitHub. [source]
$ curl -sSL https://localai.io/install.sh | sh detecting platform linux/amd64 fetching latest release binary installed to /usr/local/bin/local-ai [source]
Download the asset for your platform from the [`cli-preview-0.10.0`](https://github.com/microsoft/Foundry-Local/releases/tag/cli-preview-0.10.0) GitHub release. [source]

Facts and statements

Ollama can run any public (and, with an Ollama SSH key added to the Hub account, private) GGUF from the Hugging Face Hub using `ollama run hf.co/{username}/{repository}`; `huggingface.co` works as the domain too. [source]
Ollama's registry typically offers only a few quants (commonly Q4_K_M and Q8_0), and Ollama's own quantize supports only Q4_K_S, Q4_K_M, Q8_0, F16, F32 per issues cited by a critic; Hub repos expose the full llama.cpp quant range (e.g. IQ3_M) via the `:quant` tag. [source]
For Hub pulls, Ollama picks a chat template from the GGUF's `tokenizer.chat_template` metadata. A repo can override with files named `template` (Go template, not Jinja), `system`, and `params` (JSON). [source]
A quantization is selected by appending a tag (`:Q8_0`, `:IQ3_M`, case-insensitive, or a full filename). With no tag, Ollama uses Q4_K_M if present in the repo, otherwise one reasonable quant from the repo. [source]
Models are pulled from Docker Hub, any OCI-compliant registry, or Hugging Face; GGUF and Safetensors files can be packaged as OCI artifacts and published to any registry. [source]
Each Ollama library model exposes its upstream license text as a blob; the llama3.2 blob shows the Llama 3.2 Community License Agreement (version date 2024-09-25), so the upstream license, not an Ollama license, governs the weights. [source]
Foundry Local is Microsoft's on-device AI inference tool — an end-to-end local AI solution for building applications that run entirely on a user's device. Its GitHub repository describes it as bringing native SDKs, automatic hardware acceleration, and a curated model catalog together in a lightweight package aimed at developers who need inference to happen on the device itself, not on a remote server. [source]
Docker Model Runner intelligently routes your request: if you pull a GGUF model, it utilizes llama.cpp; if you pull a safetensors model, it leverages the power of vLLM. With Docker Model Runner, both can be pushed and pulled as OCI images to any OCI registry. [source]
Model formatGGUF (flexible quantization)GGUF (flexible quantization)ONNX (from Foundry Catalog) [source]
DMR supports three inference engines: llama.cpp (default, GGUF, all platforms), vLLM (Safetensors, NVIDIA GPUs; Linux x86_64 and Windows WSL2) and Diffusers (image generation, NVIDIA on Linux). [source]
Engine routing is automatic by model format: a GGUF model uses llama.cpp, a Safetensors model uses vLLM. The vLLM backend is installed once with 'docker model install-runner --backend vllm --gpu cuda'. [source]
| Engine | Best for | Model format | |--------|----------|--------------| | [llama.cpp](/ai/model-runner/inference-engines/#llamacpp) | Local development, resource efficiency | GGUF (quantized) | | [vLLM](/ai/model-runner/inference-engines/#vllm) | Production, high throughput | Safetensors | | [Diffusers](/ai/model-runner/inference-engines/#diffusers) | Image generation (Stable Diffusion) | Safetensors | [source]
The ollama:// scheme lets LocalAI pull models from the Ollama registry, so Ollama-library content is reachable without running Ollama. [source]
Hugging Face Hub supports all file formats, but has built-in features for [GGUF format](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md), a binary format that is optimized for quick loading and saving of models, making it highly efficient for inference purposes. GGUF is designed for use with GGML and other executors. GGUF was developed by [@ggerganov](https://huggingface.co/ggerganov) who is also the developer of [llama.cpp](https://github.com/ggerganov/llama.cpp), a popular C/C++ LLM inference framework. Models initially developed in frameworks like PyTorch can be converted to GGUF [source]
File Format Support: Primarily GGUF format with all quantization levels (Q2_K through Q8_0). Automatic conversion from Hugging Face models available through Modelfile creation. For efficient storage management, you may need to move Ollama models to a different drive or folder. [source]
Foundry Local is Microsoft's on-device AI inference tool — native SDKs for C#, JavaScript, Rust, and Python, automatic hardware-acceleration detection, and a curated model catalog — and it is a separate product from the cloud-based Azure AI Foundry. [source]
The problems with this approach compound quickly. Ollama only auto-detects chat templates it already knows about from a hardcoded list. If a GGUF file has a valid Jinja chat template embedded in its metadata but it doesn’t match one of Ollama’s known templates, Ollama falls back to a bare {{ .Prompt }} template, silently breaking the model’s instruction format. The user has to manually extract the chat template from the GGUF, translate it into Go template syntax (which is different from Jinja), and write it into a Modelfile. Meanwhile, llama.cpp reads the embedded template and just uses it. [source]
local-ai run huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf [source]
wget https://huggingface.co/TheBloke/phi-2-GGUF/resolve/main/phi-2.Q4_K_M.gguf \ -O models/phi-2.Q4_K_M.gguf [source]
Hugging Face Statistics (2026): 60+ Data Points on Model Downloads, Open-Weight Families, and Hub Security [source]
DMR partners with Google, Hugging Face, and VMware Tanzu to expand the AI model ecosystem available through Docker Hub. If you’re new to Docker or need a refresher on Docker commands, our Docker Cheatsheet provides a comprehensive guide to essential Docker operations. [source]
- https://github.com/ollama/ollama/blob/main/docs/README.md - https://huggingface.co/docs/hub/en/gguf [source]
- **Curated model catalog** — A catalog of high-quality models optimized for on-device use across a wide range of consumer hardware. The catalog covers chat completions (for example, GPT OSS, Qwen, DeepSeek, Mistral and Phi) and audio transcription (for example, Whisper). Every model goes through extensive quantization and compression to deliver the best balance of quality and performance. Models are versioned, so your application can pin to a specific version or automatically receive updates. [source]
Foundry Local (github.com/microsoft/Foundry-Local) is Microsoft's on-device AI inference tool — an end-to-end local AI solution for building applications that run entirely on a user's device, with native SDKs for C#, JavaScript, Rust, and Python. [source]
Supported platforms are Windows, macOS (Apple Silicon) and Linux. An Azure subscription is not required. Prompts and outputs stay on-device; the network is used only for model/execution-provider downloads and optional diagnostics. [source]
wget https://huggingface.co/TheBloke/Luna-AI-Llama2-Uncensored-GGUF/resolve/main/luna-ai-llama2-uncensored.Q4_0.gguf -O models/luna-ai-llama2 [source]
Docker Model Runner: Leverages Docker’s security model with container isolation. Models packaged as OCI Artifacts can be scanned and signed. Distribution through Docker Hub enables access control and vulnerability scanning for enterprise users. [source]
LocalAI Multimodal AI, flexibility ⭐⭐⭐⭐⭐ Stable ✅ Full Web UI GGUF, PyTorch, GPTQ, AWQ, Safetensors NVIDIA, AMD, Apple ✅ Yes [source]
→Curated model catalog includes model families such as Qwen, DeepSeek, Mistral, Phi, and GPT-OSS, plus Whisper for audio transcription; each downloaded model carries its own license [source]
Curated model catalog includes model families such as Qwen, DeepSeek, Mistral, Phi, and GPT-OSS, plus Whisper for audio transcription — each downloaded model carries its own separate license terms [source]
Foundry Local's catalog is curated (model families such as Qwen, DeepSeek, Mistral, Phi, GPT-OSS, plus Whisper), not an open upload system. Check the catalog for a specific model before assuming it is available, and remember each model carries its own separate license terms. [source]
Key Features: Simple model management with commands like ollama run llama3.2, OpenAI-compatible API for drop-in replacement of cloud services, extensive model library supporting Llama, Mistral, Gemma, Phi, Qwen and others, structured outputs capability, and custom model creation via Modelfiles. [source]
- **OpenAI-compatible API** — Supports OpenAI request and response formats including the [OpenAI Responses API format](https://developers.openai.com/api/reference/resources/responses). If your application already uses the OpenAI SDK, point it to a Foundry Local endpoint with minimal code changes. [source]
GGUF, the model format created by Georgi Gerganov, was designed with one core principle: single-file deployment. Bullet point #1 in the GGUF spec reads: “Full information: all information needed to load a model is contained in the model file, and no additional information needs to be provided by the user.” Chat templates, stop tokens, model metadata, it’s all embedded in the file. You point llama.cpp at a GGUF and it works. [source]
Docker Model Runner Container workflows ⭐⭐⭐ Alpha/Beta ⚠️ Limited Docker Desktop GGUF (depends) NVIDIA, AMD Partial [source]
Today, we’re excited to announce that Docker Model Runner now integrates the vLLM inference engine and safetensors models, unlocking high-throughput AI inference with the same Docker tooling you already use. [source]
GGUF (GPT-Generated Unified Format): The native format for llama.cpp, GGUF is designed for high portability and quantization. It is excellent for running models on commodity hardware where memory bandwidth is limited. It packages the model architecture and weights into a single file. [source]
Repositories under mlx-community are imported with the native MLX backend. LocalAI uses Hugging Face’s pipeline metadata to select mlx-vlm for vision-language models and mlx-audio for text-to-speech models; other MLX repositories use mlx. An explicit backend selection in the import form always overrides this automatic routing. [source]
Ollama added the Modelfile on top of this. It’s a separate configuration file, inspired by Dockerfiles, naturally, that specifies the base model, chat template, system prompt, sampling parameters, and stop tokens. Most of this information already exists inside the GGUF file. As one Hacker News commenter put it: “We literally just got rid of that multi-file chaos only for Ollama to add it back.” [source]
Docker Model Runner: Launched April 2025 with strong enterprise backing. Partnerships with Google, Hugging Face, and VMware Tanzu AI Solutions ensure broad model availability. Integration with Docker’s massive developer community (millions of users) provides instant ecosystem access. Still building community-specific resources as a new product. [source]
File Format Support: Depends on connected backends (typically GGUF via Ollama/LocalAI). [source]
Docker Model Runner — a CLI-and-API feature bundled with Docker Desktop/Engine for developers already working in a Docker-based workflow, another option to weigh against Foundry Local's native-SDK approach if your stack is container-centric. See the full Docker Model Runner review. [source]
For production deployments, LocalAI can route requests to external high-performance backends like vLLM while maintaining the unified OpenAI-compatible API. [source]
Ollama was founded in 2021 by Jeffrey Morgan and Michael Chiang, both previously behind Kitematic, a Docker GUI that was acquired by Docker Inc. They went through Y Combinator’s Winter 2021 batch, raised pre-seed funding, and launched publicly in 2023. From day one, the pitch was “Docker for LLMs”, a convenient wrapper that downloads and runs models with a single command. Under the hood, it was llama.cpp doing all the work. [source]
Model Distribution Docker Hub, OCI registries Ollama registry [source]
Model Library: Curated repository at ollama.com/library [source]
Model flexibility: Works seamlessly with popular open-weight models like GPT-OSS, Qwen3, Mistral, Llama 3, and others in the safetensors format. [source]
Model freedom mattersThe GGUF ecosystem (Ollama / llama.cpp) offers far more choices [source]
* Allowing users to filter models at https://huggingface.co/models. * Displaying the model's license. * Adding datasets to the metadata will add a message reading `Datasets used to train:` to your model page and link the relevant datasets, if they're available on the Hub. [source]
Dataset and language identifiers are those listed on the [Datasets](https://huggingface.co/datasets) and [Languages](https://huggingface.co/languages) pages. [source]
Note: The galleries available in LocalAI can be customized to point to a different URL or a local directory. For more information on how to setup your own gallery, see the Gallery Documentation. [source]
huggingface://repository_id/model_file (e.g., huggingface://TheBloke/phi-2-GGUF/phi-2.Q8_0.gguf) [source]
For over a year, Ollama’s README contained no mention of llama.cpp. Not in the README, not on the website, not in their marketing materials. The project’s binary distributions didn’t include the required MIT license notice for the llama.cpp code they were shipping. This isn’t a matter of open-source etiquette, the MIT license has exactly one major requirement: include the copyright notice. Ollama didn’t. [source]
File Formats: GGUF, GGML, Safetensors, PyTorch, GPTQ, AWQ [source]
LocalAI’s model gallery provides pre-configured models, but the platform also supports custom configurations for specialized deployments. This flexibility serves teams requiring fine-tuned or specialized models across different modalities. [source]
If you’re an app developer who wants to embed local inference into your product, you’re stuck with two options: tell users “please install Ollama first” in your README, or link ONNX Runtime or llama.cpp directly and implement model management yourself. [source]
On first launch, the optimal variant for the machine’s hardware is automatically downloaded from the Foundry Catalog and cached locally. [source]
Once you have created your `Modelfile`, use the `ollama create` command to build the model. [source]
LocalAI auto-detects GPU capability (NVIDIA CUDA 12/13, AMD ROCm, Intel oneAPI, Vulkan, Apple Silicon, Jetson) and downloads the matching backend. CPU-only operation is supported. [source]
> **Automatic Backend Detection**: LocalAI automatically detects your GPU capabilities and downloads the appropriate backend. For advanced options, see [GPU Acceleration](https://localai.io/features/gpu-acceleration/). [source]
In mid-2025, Ollama followed through on that distancing. They moved away from using llama.cpp as their inference backend and built a custom implementation directly on top of ggml, the lower-level tensor library that llama.cpp itself uses. Their stated reason was stability, llama.cpp moves fast and breaks things, and Ollama’s enterprise partners need reliability. [source]
Docker Model Runner uses native docker model pull and docker model run commands, packaging models as OCI artifacts. [source]
Optimized Performance: Built on llama.cpp, Ollama is highly optimized for inference speed with quantization support (Q4, Q5, Q8). [source]
* We first began by pulling all Hugging Face model's on the hub and, in particular, subsections on Limitations and Bias ("Risks" subsections were largely not present). * Based on inputs that were the most continuously used with a higher number of model downloads, grouped by model typed, the tool provides prompted text within the Bias, Risks and Limitations sections. We also prompt a default text if the model type is not specified. [source]
We've also created a javascript GGUF parser that works on remotely hosted files (e.g. Hugging Face Hub). [source]
Foundry Local may collect usage data and send it to Microsoft to help improve our products and services. See the [privacy statement](sdk_v2/cpp/docs/Privacy.md) for more details. [source]
- **Automatic hardware acceleration** — Foundry Local detects the available hardware on the user's device and selects the best execution provider and device (NPU, GPU or CPU). [source]
- **Smart model management** — Foundry Local handles the full lifecycle of models on end-user devices. Models download automatically on first use, are cached locally for instant subsequent launches, and the best-performing variant is selected for the user's specific hardware. [source]
Now run the `ollama create` command from the directory where you created the `Modelfile`: [source]
You are able to add a license to any repo that you create on the Hugging Face Hub to let other users know about the permissions that you want to attribute to your code or data. The license can be specified in your repository's `README.md` file, known as a _card_ on the Hub, in the card's metadata section. Remember to seek out and respect a project's license if you're considering using their code or data. [source]
local-ai run huggingface://TheBloke/phi-2-GGUF [source]
Foundry Local SDK is licensed under the MIT license. For more details, see the [LICENSE](LICENSE) file. Foundry Local CLI is licensed under the Microsoft Software License Terms. For more details, read the [LICENSE](LICENSE) file. [source]
Bring your own ONNX Runtime GenAI models into Foundry Local without copying, downloading, or transferring ownership of the model files. [source]
4 Ongoing — curated model catalog expansion [source]
You want a single, fully open-source license across the entire product — Foundry Local's CLI carries Microsoft's own proprietary license terms, not MIT, even though the SDK is MIT-licensed [source]
Foundry Local uses ONNX Runtime for inference, with automatic hardware-acceleration detection that selects the optimal execution provider — NPU, GPU, or CPU — per device. [source]
For C#, running dotnet add package pulls in the NuGet package, and at build time, the Foundry Local Core native libraries (.dll / .so / .dylib) and ONNX Runtime binaries are copied into the output directory. [source]
graph TD A[Application] --> B[Foundry Local SDK Python / C# / JS / Rust] B --> C[Foundry Local Core Native Library .dll / .so / .dylib] C --> D[ONNX Runtime] C --> E[Model Manager Download, Cache, Load] D --> F{Hardware Detection} F -->|Windows| G[Windows ML GPU / NPU / CPU auto-select] F -->|macOS| H[Metal Apple Silicon GPU] F -->|Linux| I[CPU fallback] [source]
How to fill out each section of the model card is described in [the Annotated Model Card](https://huggingface.co/docs/hub/model-card-annotated). [source]
Our work presents a view of where we think model cards stand right now and where they could go in the future, at Hugging Face and beyond. This work is a “snapshot” of the current state of model cards, informed by a landscape analysis of the many ways ML documentation artefacts have been instantiated. It represents one perspective amongst multiple about both the current state and more aspirational visions of model cards. In this blog post, we summarise our work, including a discussion of the broader, growing landscape of ML documentation tools, the diverse audiences for and opinions about model [source]
3) A [Model Card Creator Tool](https://huggingface.co/spaces/huggingface/Model_Cards_Writing_Tool), to ease card creation without needing to program, and to help teams share the work of different sections. [source]
4) A [User Study](./model-cards-user-studies) on Model Card usage at Hugging Face [source]
**Please cite as:** Ozoani, Ezi and Gerchick, Marissa and Mitchell, Margaret. Model Card Guidebook. Hugging Face, 2022. https://huggingface.co/docs/hub/en/model-card-guidebook [source]
Foundry Local exposes OpenAI request/response formats, including the Responses API, via an optional OpenAI-compatible local web server. Existing OpenAI-SDK code can point at the endpoint with minimal changes. [source]
Context: RedMonk’s license scan finds Apache-2.0 has about 2.5 times more licensed projects than MIT, and the Economies of Open Intelligence paper found open-weights models surpassed truly open source models (with open training data) for the first time in 2025. Open weights and open source are diverging. [source]
Most backends wrap a best-in-class upstream engine. A handful of them are native C/C++/GGML engines (no Python at inference) developed and maintained by the LocalAI project itself: [source]
For production and enterprise: Both DMR and third-party solutions (vLLM, TGI) in Docker offer orchestration, monitoring, and scalability [source]
- On Linux, Docker Model Runner and its inference engines, such as Diffusers, run inside a container, which provides the isolation boundary. - On macOS and Windows, the engines don't run inside a container, so Docker Model Runner runs them in a sandboxed environment (seatbelt/sandbox-exec and Job Objects respectively) [source]
LocalAI: OpenAI-Compatible Local LLM Server with Multimodal Support [source]
API Maturity: Highly mature as full OpenAI drop-in replacement supporting all OpenAI endpoints plus additional features. Includes full streaming support, native function calling via OpenAI-compatible tools API, image generation and processing, audio transcription (Whisper), text-to-speech, configurable rate limiting, and built-in API key authentication. LocalAI excels at tasks like converting HTML content to Markdown using LLM thanks to its versatile API support. [source]
LocalAI takes a different approach as a comprehensive OpenAI-compatible API gateway supporting text, image, audio, and video generation. ‘ [source]
LocalAI’s flexibility enables complex deployments where a single OpenAI-compatible endpoint routes to multiple specialized backends. This architecture serves teams requiring specialized or fine-tuned models across different modalities. [source]
Docker Model Runner Integrates vLLM for High-Throughput Inference [source]
By bringing vLLM to Docker Model Runner, we’re bridging the gap between fast local experimentation and robust production inference. [source]
Running vLLM models with Docker Model Runner is as simple as installing the backend and running your model, no special setup required. [source]
This flexibility makes Docker Model Runner a first in the industry — no other tool lets you switch between multiple inference engines within a single, portable, containerized workflow. [source]
For development and debugging, there’s also a mode that starts Foundry Local Core as an OpenAI-compatible HTTP server. [source]
Foundry Local provides an OpenAI-compatible API with chat completions and transcription endpoints. [source]
By default, a template will be selected automatically from a list of commonly used templates. It will be selected based on the built-in `tokenizer.chat_template` metadata stored inside the GGUF file. [source]
*if there's any inaccuracy on the table above, please open a PR on [this file](https://github.com/huggingface/huggingface.js/blob/main/packages/gguf/src/quant-descriptions.ts).* [source]
Foundry Local powered by Azure Local brings models and agentic AI — including RAG, chat — to customer-owned distributed infrastructure. This is in preview now, with more coming soon. [source]
An OpenAI-compatible API in front of llama.cpp, so a laptop could answer the same calls as the cloud. [source]
API Maturity: Alpha/Beta stage with evolving APIs. Container-native interfaces with underlying engine determining specific capabilities (usually based on GGUF/Ollama). [source]
Home/Power Local LLM/Foundry Local Review: Microsoft's On-Device AI Runtime [source]
Overview & Reference Foundry Local Review: Microsoft's On-Device AI Runtime [source]
→On-device AI inference tool from Microsoft; source at github.com/microsoft/Foundry-Local, project site foundrylocal.ai [source]
→Distinct from Azure AI Foundry (Microsoft's cloud AI platform) — Foundry Local runs on-device with no Azure subscription required for local-only use [source]
Foundry Local is Microsoft's on-device AI inference tool; source at github.com/microsoft/Foundry-Local, project site foundrylocal.ai [source]
Foundry Local lets a developer build an app that runs AI models directly on a user's Windows, Mac, or Linux machine, picking whichever chip (NPU, GPU, or CPU) does the job fastest on that specific device, without needing a cloud account for local use. It is not the same thing as Azure AI Foundry, which is Microsoft's cloud platform — the two can work together, but Foundry Local's whole point is running offline, on-device. [source]
Foundry Local's feature set centers on making on-device inference straightforward to integrate for a developer, per its GitHub README. [source]
Licensing is split: the Foundry Local SDK is MIT, the CLI is under Microsoft proprietary terms, and each catalog model carries its own license. Ollama is fully MIT. [source]
Command Style docker model pull/run/package ollama pull/run/list [source]
Convert model references: ollama run llama2 → docker model pull ai/llama2 and docker model run ai/llama2 [source]
The introduction of DMR narrows the gap between Docker and Ollama in terms of ease of use. Ollama still wins on simplicity for quick prototyping, while DMR excels for teams already invested in Docker workflows. Both approaches are actively developed, production-ready, and the ecosystem is mature enough that switching between them is relatively painless. [source]
API Maturity: Stable for connecting to existing installations. No separate server required as it extends functionality of other tools like Ollama and LocalAI. [source]
> For the full CLI reference and advanced usage, see the [CLI documentation on Microsoft Learn](https://learn.microsoft.com/en-us/azure/foundry-local/reference/reference-cli). [source]
- [Foundry Local Documentation](https://learn.microsoft.com/en-us/azure/foundry-local/) on Microsoft Learn - [Foundry Local Lab](https://github.com/Microsoft-foundry/foundry-local-lab) — Hands-on exercises and step-by-step instructions [source]
An open weights function calling model based on Llama 3, competitive with GPT-4o function calling capabilities. [source]
If a developer specifically wants automatic hardware-acceleration selection across NPU, GPU, and CPU with native C#, JavaScript, Rust, or Python SDKs — for example, building a Windows or cross-platform client application — Foundry Local's design fits that use case directly. If a developer wants the broadest model library and a fully open-source runtime, Ollama's workflow is the more direct fit. See the full Ollama review for details. [source]
The catalog is curated and includes model families such as Qwen, DeepSeek, Mistral, Phi, and GPT-OSS, plus Whisper for audio transcription. Each model carries its own separate license terms. [source]
Understanding the architectural differences between Ollama and LocalAI reveals why each platform excels in different scenarios. [source]
The multimodal story strongly differentiates these platforms. While Ollama focuses primarily on text generation, LocalAI provides comprehensive support for multiple modalities. [source]
LangChain: Native integrations for both Ollama and LocalAI [source]
For new projects exploring local LLMs, Ollama provides the fastest path to running models. For organizations with existing OpenAI-based applications or multimodal requirements, LocalAI’s API compatibility reduces migration effort. [source]
Many teams benefit from using both: Ollama for development and exploration, LocalAI for production deployments requiring advanced features. [source]
Hire vetted remote AI developers with Second Talent to deploy secure local LLM infrastructure using Ollama, LocalAI, or enterprise AI platforms. [source]
graph TD subgraph "Ollama approach" A1[User installs Ollama] --> A2[Starts ollama serve] A2 --> A3[App connects via HTTP] end subgraph "Foundry Local approach" B1[Developer adds SDK package] --> B2[Build bundles native library] B2 --> B3[User installs the app] B3 --> B4[AI inference ready immediately] end [source]
More models can be found on the [Ollama library](https://ollama.com/library). [source]
[1]: https://ollama.com/library [source]
- **Lightweight runtime** — The runtime handles model acquisition, hardware acceleration, model management, and inference (via [ONNX Runtime](https://onnxruntime.ai/)). [source]
name: phi-2 parameters: model: phi-2.Q4_K_M.gguf temperature: 0.7 context_size: 2048 threads: 4 backend: llama-cpp [source]
Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates. [source]
Ollama gained traction by being the first easy llama.cpp wrapper, then spent years dodging attribution, misleading users, and pivoting to cloud, all while riding VC money earned on someone else's engine. Here's the full history, and why the alternatives are better. [source]
The response to the PR was revealing. Ollama’s team wrote: “We spend a large chunk of time fixing and patching it up to ensure a smooth experience for Ollama users… Overtime, we will be transitioning to more systematically built engines.” Translation: we’re not going to give llama.cpp prominent credit, and we plan to distance ourselves from it anyway. [source]
As one Hacker News commenter put it: “I’m continually puzzled by their approach, it’s such self-inflicted negative PR. Building on llama is perfectly valid and they’re adding value on ease of use here. Just give the llama team proper credit.” Another: “The fact that Ollama has been downplaying their reliance on llama.cpp has been known in the local LLM community for a long time.” [source]
Red Hat’s ramalama is worth a look too, a container-native model runner that explicitly credits its upstream dependencies front and center. Exactly what Ollama should have done from the start. [source]
The local LLM ecosystem doesn’t need Ollama. It needs llama.cpp. The rest is packaging, and better packaging already exists. [source]
Ollama has emerged as one of the most popular tools for local LLM deployment, particularly among developers who appreciate its command-line interface and efficiency. Built on top of llama.cpp, it delivers excellent token-per-second throughput with intelligent memory management and efficient GPU acceleration for NVIDIA (CUDA), Apple Silicon (Metal), and AMD (ROCm) GPUs. [source]
Technical Comparison With Ollama and llama.cpp [source]
The GGUF file location should be specified as an absolute path or relative to the `Modelfile` location. [source]
iv. Your use of the Llama Materials must comply with applicable laws and regulations (including trade compliance laws and regulations) and adhere to the Acceptable Use Policy for the Llama Materials (available at https://developer.meta.com/ai/llama4/use-policy), which is hereby incorporated by reference into this Agreement. [source]
LLAMA 3.2 COMMUNITY LICENSE AGREEMENT [source]
a copy of this Agreement with any such Llama Materials; and (B) prominently display “Built with Llama” [source]
Exposes an OpenAI-compatible API (including OpenAI Responses API request/response formats) through a local web server, and uses ONNX Runtime for inference [source]
3 Ongoing — ONNX Runtime as the inference engine [source]
Safetensors: The native format for vLLM and the modern standard for high-end inference, safetensors is built for high-throughput performance. [source]
Gemma 4: Expanding the Gemmaverse with Apache 2.0 [source]
We are disheartened to see a couple of individuals entangling their criticisms of the Open Source AI Definition with Meta’s abuse of the term Open Source referring to Llama. The OSAID did not create any confusion about the proprietary nature of Llama; in fact, Meta’s open washing, and our response to it, predate the OSAID entirely. [source]
* `OLLAMA_KV_CACHE_TYPE` - The quantization type for the K/V cache. Default is `f16`. [source]
* `f16` - high precision and memory usage (default). * `q8_0` - 8-bit quantization, uses approximately 1/2 the memory of `f16` with a very small loss in precision, this usually has no noticeable impact on the model's quality (recommended if not using f16). * `q4_0` - 4-bit quantization, uses approximately 1/4 the memory of `f16` with a small-medium loss in precision that may be more noticeable at higher context sizes. [source]
You may need to experiment with different quantization types to find the best balance between memory usage and quality. [source]
Foundry Local detects GPU, NPU and CPU and picks an execution provider automatically, falling back to CPU. On Windows it uses WinML; on macOS it uses the Apple Silicon GPU via Metal. [source]
#statistics#hugging-face#open-source-ai#open-weight-models#2026 [source]
Public models on the Hub (Aug 22, 2026)3,012,377Ivan Fioravanti, Three Million Models and Counting, Hugging Face blog 2026 [source]
Days to add the 2nd and 3rd million models335 and 349Ivan Fioravanti, Hugging Face blog 2026 [source]
Hugging Face users (2025)13 millionHugging Face, State of Open Source: Spring 2026 [source]
Public models (Aug 22, 2026)3,012,377Ivan Fioravanti, Hugging Face blog 2026 [source]
Hugging Face, State of Open Source on Hugging Face: Spring 2026 (published March 17, 2026) [source]
Ivan Fioravanti, Three Million Models and Counting, Hugging Face blog (community post compiling Hub API counts, August 2026) [source]
RedMonk, License Distribution on Hugging Face (May 12, 2026) [source]
NBC News, OpenAI agents hacked Hugging Face, investigations find (August 26, 2026) [source]
AI Forensics, Unmoderated by Design: How Hugging Face Enables NCII (July 28, 2026) [source]
Created by [Ettore Di Giacinto](https://github.com/mudler) and maintained by the [LocalAI team](#team). [source]
https://huggingface.co/docs/hub/moderation.md [source]
Whether you’re building a desktop assistant, a healthcare decision-support tool, a private coding companion, or an offline-capable edge application, Foundry Local gives you production-grade AI that runs entirely on the user’s machine. [source]
Microsoft Foundry spans cloud to edge — from Microsoft Foundry in the cloud for frontier models, agents, and fine-tuning, to Foundry Local for on-premises and distributed deployments validated on Azure Local, to Foundry Local running natively across Windows, MacOS, Android, and other devices including phones, laptops, and desktops. [source]
Foundry Local is cross platform: Windows, Linux and macOS. On Windows it integrates with Windows ML (WinML) for inferencing on Windows and to acquire hardware matched execution provider plugins from the OS / Windows update, which ensures driver compatibility and version negotiation. Your end users do not need to be concerned with installing device drivers to get the most optimal performance. On macOS Foundry Local runs natively on the Apple Silicon GPU through Metal. [source]
Samples for each language are provided in the Foundry Local GitHub repository. [source]
This section covers everything you need to know about installing and configuring models in LocalAI. You’ll learn multiple methods to get models running. [source]
local-ai run oci://localai/phi-2:latest [source]
docker run -p 8080:8080 -v $PWD/models:/models -ti --rm localai/localai:latest --models-path /models --context-size 700 --threads 4 [source]
local-ai backends list local-ai backends install llama-cpp # if needed [source]
docker model run ai/smollm2 "Explain Docker Model Runner" [source]
For a full reference of docker model commands, packaging options, configuration flags, and practical examples, see our detailed Docker Model Runner Cheatsheet: Commands & Examples. It covers model pulling, packaging, configuration, and best practices for deploying LLMs locally with Docker. [source]
The landscape of local LLM deployment has evolved significantly with Docker’s introduction of Docker Model Runner (DMR) in 2025. The choice now depends on your specific requirements: [source]
Docker Model Runner Cheatsheet: Commands & Examples [source]
Whether you're building generative AI applications, experimenting with machine learning workflows, or integrating AI into your software development lifecycle, Docker Model Runner provides a consistent, secure, and efficient way to work with AI models locally. [source]
What landed in LocalAI 4.11 [source]
$ podman run -p 8080:8080 --name local-ai -ti localai/localai:latest [source]
$ kubectl apply -f https://localai.io/install/kubernetes.yaml deployment.apps/local-ai created service/local-ai created [source]
Ollama Developers, API integration ⭐⭐⭐⭐⭐ Stable ❌ Limited 3rd party GGUF NVIDIA, AMD, Apple ✅ Yes [source]
LocalAI positions itself as a comprehensive AI stack, going beyond just text generation to support multimodal AI applications including text, image, and audio generation. [source]
Docker Model Runner is Docker’s relatively new entry into local LLM deployment, leveraging Docker’s containerization strengths with native integration, Docker Compose support for easy multi-container deployments, simplified volume management for model storage and caching, and container-native service discovery. [source]
File Format Support: Depends on backend being used (typically GGUF via Ollama). [source]
Foundry Local supports Windows, macOS (Apple silicon), and Linux. [source]
Canonical repository: github.com/microsoft/Foundry-Local; project site: foundrylocal.ai [source]
6 On supported hardware, Foundry Local's automatic hardware-acceleration detection selects the NPU, GPU, or CPU for you; no manual execution-provider configuration is required for the default path. [source]
Most confusion about Foundry Local comes from conflating it with Azure AI Foundry, or assuming the whole product is open source. [source]
https://huggingface.co/docs/hub/security-pickle.md [source]
Foundry Local runs on your users' own devices, so your app works offline and you don't need an Azure subscription. If you need enterprise-scale AI inference on your own infrastructure with Kubernetes-native operations and Azure Arc management instead, see Foundry Local on Azure Local. [source]
No. Foundry Local runs entirely on local hardware. No Azure subscription is required. [source]
This comprehensive multimodal support makes LocalAI suitable for applications requiring multiple AI capabilities through a single deployment. [source]
LocalAI provides full OpenAI API compatibility, enabling teams to switch from OpenAI to local models with minimal code changes. Simply update the base URL and API key. This compatibility extends to: [source]
For multi-tenant SaaS platforms, LocalAI’s model management and resource isolation capabilities provide flexibility. Teams can route different tenants or use cases to optimized backends while maintaining a consistent API surface. [source]
LocalAI’s comprehensive OpenAI compatibility and multimodal support serve teams migrating from cloud APIs or building complex AI applications. [source]
We’re incredibly excited about this new chapter for Docker Model Runner, and we can’t wait to see what we can build together. Let’s get to work! [source]
Foundry Local delegates this choice to Windows ML (on Windows) or Metal (on macOS), shielding developers from hardware differences. [source]
Foundry Local delegates hardware selection to Windows ML, so manual tuning isn’t available. [source]
These Terms of Service (“Terms”) govern your access to and use of the services provided by Ollama Inc. (“Ollama,” “we,” “us,” or “our”), including our website, software, APIs, and related services (the “Service”). By accessing or using the Service, you agree to be bound by these Terms and our Privacy Policy, which is incorporated into these Terms by reference. If you do not agree, do not use our services. [source]
The format is: huggingface://<repository>/<model-file> ( is optional) [source]
Method 1: Using the Model Gallery (Easiest) [source]
The same Models page owns the complete lifecycle. Switch to Installed to search local configurations, filter them by running, idle, disabled, pinned, or distributed state, and open a model’s runtime controls. Load and stop are on each row; edit, pin, disable, backend logs, and remove are in the row menu and in the details of the selected model. The current view, search, filter, and selection are stored in the URL so links and browser history preserve your place. The disk strip in the page header shows the free space on the models disk and opens a review of what can be removed to free more (see [source]
GGUF: Modern format, recommended for most use cases [source]
From Files and versions tab on a model page, open GGUF viewer on a particular GGUF file. [source]
Code: The complete source code used to train and run the system. The Code shall represent the full specification of how the data was processed and filtered, and how the training was done. Code shall be made available under OSI-approved licenses. [source]
These numbers come from ONNX-optimized models, though — not a direct comparison with GGUF quantized models. Take them with appropriate context. [source]
For a split GGUF model, keep the original split filenames and use a wildcard that matches every shard. [source]

Related concepts

Ollama — is a related of Ollama alternatives and model sourcing
Hugging Face Hub — is a hyponym of Ollama alternatives and model sourcing
LocalAI — is a hyponym of Ollama alternatives and model sourcing
Foundry Local — is a hyponym of Ollama alternatives and model sourcing
Llama — is a instance of Ollama alternatives and model sourcing
GGUF — is a part of Ollama alternatives and model sourcing
inference engine — is a part of Ollama alternatives and model sourcing
Docker Model Runner — is a hyponym of Ollama alternatives and model sourcing
pull counts — is a measure of Ollama alternatives and model sourcing
quantization — is a part of Ollama alternatives and model sourcing
on-device — is a related of Ollama alternatives and model sourcing
Ollama library — is a hyponym of Ollama alternatives and model sourcing
open-weight — is a synonym of Ollama alternatives and model sourcing
GGUF publisher — is a instance of Ollama alternatives and model sourcing
Safetensors — is a part of Ollama alternatives and model sourcing
OpenAI-compatible — is a part of Ollama alternatives and model sourcing
curated catalog — is a part of Ollama alternatives and model sourcing
Gemma — is a instance of Ollama alternatives and model sourcing
model card — is a part of Ollama alternatives and model sourcing
ONNX — is a part of Ollama alternatives and model sourcing

Children

DMR vLLM backend (frontier)
Docker Hub ai/ namespace (frontier)
Docker Model Runner
Foundry Local on Azure Local (frontier)
GGUF quantization levels and publisher quality (frontier)
Llama Acceptable Use Policy (frontier)
LocalAI
LocalAI Ollama API compatibility (frontier)
LocalAI backend gallery and OCI images (frontier)
LocalAI distributed mode (frontier)
Microsoft Foundry Local
Model-output training restrictions (frontier)
ONNX Runtime GenAI and Olive model conversion (frontier)
Ollama Modelfile and Go template conversion (frontier)
Open-weight model catalogs: Ollama library versus Hugging Face Hub
Open-weight model licenses and Ollama library terms
Qwen and Mistral license variants (frontier)
Windows ML execution providers (frontier)
docker model package (OCI model artifacts) (frontier)
llama.cpp -hf direct Hub pulls (frontier)

Frontier under this node: DMR vLLM backend, Docker Hub ai/ namespace, Foundry Local on Azure Local, GGUF quantization levels and publisher quality, Llama Acceptable Use Policy, LocalAI Ollama API compatibility, LocalAI backend gallery and OCI images, LocalAI distributed mode, Model-output training restrictions, ONNX Runtime GenAI and Olive model conversion, Ollama Modelfile and Go template conversion, Qwen and Mistral license variants, Windows ML execution providers, docker model package (OCI model artifacts), llama.cpp -hf direct Hub pulls

← the whole tree · 3D view· how to read this page