Ollama core platform

Parent: Ollama · Published reference · snapshot 2026-10-14 · skill ollama-expert/references/ollama-core-platform.md

Also known as: Ollama core platform (cohort A)

↓ Facts as markdown all context files

Ollama is a local LLM runtime whose core platform is a native REST API (/api/generate, /api/chat with NDJSON streaming, errors and usage fields), a Modelfile build format (FROM, PARAMETER, TEMPLATE, SYSTEM, ADAPTER), a Docker-v2-style registry with content-addressed blobs and manifests, GGUF/safetensors import and quantization tags, LoRA adapters, OpenAI and Anthropic compatible endpoints, and a think capability. Compiled from 44 source files (cached /dr page bodies, /dr claim units and the docs.ollama.com mirror) across 20 hosts.

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Definitions

Ollama is an application based on llama.cpp to interact with LLMs directly through your computer. You can use any GGUF quants created by the community ([bartowski](https://huggingface.co/bartowski), [MaziyarPanahi](https://huggingface.co/MaziyarPanahi) and [many more](https://huggingface.co/models?pipeline_tag=text-generation&library=gguf&sort=trending)) on Hugging Face directly with Ollama, without creating a new `Modelfile`. At the time of writing there are 45K public GGUF checkpoints on the Hub, you can run any of them with a single `ollama run` command. We also provide customisations like [source]
Documented PARAMETER defaults on the Modelfile reference: num_ctx 2048, temperature 0.8, seed 0, top_k 40, top_p 0.9, min_p 0.0, repeat_last_n 64 (0 disabled, -1 = num_ctx), num_predict -1 (infinite). stop has no default and is repeated once per pattern. [source]
Only FROM is required in a Modelfile; PARAMETER, TEMPLATE, SYSTEM, ADAPTER, LICENSE, MESSAGE and REQUIRES are optional. Instructions are case-insensitive and may appear in any order. [source]
Hermes specifically is a textbook case of the /v1 problem. Hermes computes a context window for each request and sends num_ctx, but it talks to a local Ollama over the OpenAI-compatible endpoint, so Ollama ignores it and every model loads at the server default. A native /api/chat adapter is in progress upstream (hermes-agent PR #4505), and there's a community stopgap package that reroutes Hermes to the native endpoint until it lands. Until then, OLLAMA_CONTEXT_LENGTH is the fix for Hermes users, full stop. [source]
Starting from an existing model's definition is the recommended authoring path: `ollama show --modelfile <model>` prints its Modelfile including TEMPLATE and parameters. [source]
MLX runtime is not available: failed to load MLX dynamic library: you are importing safetensors on a system without MLX; convert to GGUF [source]
GGUF (GPT-Unified Format) is the binary format used by llama.cpp and Ollama to store quantized model weights for efficient CPU and GPU inference. The conversion script takes the merged 16-bit PyTorch model produced by training and converts it to a self-contained GGUF file at a target quantization level. [source]
Convert and quantize — model.save_pretrained_gguf() is called with the target quantization method; Unsloth delegates to its bundled llama.cpp binaries to perform tensor-level quantization and write the GGUF file [source]
Why BFloat16 for this comparison? BF16 is a common numerical format used during inference of many large models. It means that the model parameters are represented with 16 bits of precision. Using BF16 for all models helps us to make an apples-to-apples comparison of models in a common inference setup. This allows us to compare the inherent capabilities of the models themselves, removing variables like different hardware or optimization techniques like quantization, which we'll discuss next. [source]
Current docs: `think` accepts true, false, null (model default), or a string from the model's `thinking.values`; numbers are unsupported and unsupported names fall back to the model default. Clients discover `values` and `default` from a top-level `thinking` object in /api/show; `values: [false]` means no thinking support, and a missing `thinking` object means no thinking metadata. [source]
The store is content-addressed: every filename is the sha256 of that file's contents, and manifests/ holds the JSON that says which blobs make up which model. A real manifest, from ~/.ollama/models/manifests/registry.ollama.ai/library/gemma3/270m: [source]
messages = [{'role': 'user', 'content': "What is the temperature in New York?"}] [source]
The path it prints is the model: .../registry.ollama.ai/library/llama3.3/latest. On Windows, Select-String -Path "$env:USERPROFILE\.ollama\models\manifests\*" -Pattern "4824460d29f2" -Recurse does the same job. [source]
messages = [{'role': 'user', 'content': 'What are the current weather conditions and temperature in New York and London?'}] [source]
const messages = [{ role: 'user', content: 'What are the current weather conditions and temperature in New York and London?' }] [source]
<Note> Currently this is a global option - meaning all models will run with the specified quantization type. </Note> [source]
A Modelfile is the blueprint to create and share customized models using Ollama. [source]
* the **`Modelfile` is not case sensitive**. In the examples, uppercase instructions are used to make it easier to distinguish it from arguments. * Instructions can be in any order. In the examples, the `FROM` instruction is first to keep it easily readable. [source]
The original pretrained weights are kept frozen, which means you can have multiple lightweight and portable LoRA models for various downstream tasks built on top of them. [source]
All variants struggle on Normal incidents — this is a training data imbalance issue, not a quantization artifact [source]
SYSTEM """ You are an expert programmer. Always: - Write clean, well-commented code - Follow best practices - Explain your reasoning """ [source]
num_ctx is the bug fix. Config discipline is the solution. Fix the setting today, then go delete half your soul.md this week. [source]
If it is the machine: move the models directory to a different local disk with OLLAMA_MODELS and retry there before you touch anything else. [source]
* QwQ is the reasoning model of the Qwen series. [source]

Structure and components

The current docs.ollama.com import page covers only whole-model Safetensors and GGUF import and shows no adapter workflow. The current Modelfile reference source (docs/modelfile.mdx on main) contains no occurrence of 'adapter' or 'lora'. The only documented ADAPTER guidance found is in the older 0.9.0 guide. [source]
Ollama manifests reuse the Docker image manifest v2 schema 2 envelope (schemaVersion 2, mediaType, a config object and a layers array whose entries carry mediaType, size and sha256 digest); Ollama substitutes its own layer media types (model, template, license, params) for image layers. [source]
The models directory holds blobs/ with content-addressed files named by digest with ':' replaced by '-' (sha256-<64 hex>), and manifests/ with per-tag JSON, e.g. manifests/registry.ollama.ai/library/gemma3/270m. Blobs are shared by digest, so tags with the same weights layer share one file. [source]
Ollama's OpenAI layer implements /v1/chat/completions, /v1/completions, /v1/models, /v1/embeddings and /v1/responses. Chat supports streaming, JSON mode, seed, tools, base64 vision and reasoning control; logprobs are unsupported. Embeddings reject token-array input. [source]
Default model directory: ~/.ollama/models on macOS, /usr/share/ollama/.ollama/models on Linux with the standard installer, C:\Users\%username%\.ollama\models on Windows. Setting OLLAMA_MODELS overrides it; on Linux the ollama service user needs read/write access to the new directory. [source]
A manifest is schemaVersion 2 with Docker mediaType application/vnd.docker.distribution.manifest.v2+json, a config (application/vnd.docker.container.image.v1+json) and layers typed application/vnd.ollama.image.model, .template, .license and .params. [source]
Template syntax is model-specific: copying another model's TEMPLATE breaks output, and after replacing TEMPLATE the stop parameters must be reset to that model's special tokens. SYSTEM only takes effect if the TEMPLATE contains a .System block. [source]
{"model":"deepseek-r1", "created_at":"2025-05-29T09:35:56.836222Z", "message": {"role": "assistant", "content": "The word \"strawberry\" contains **three** instances of the letter 'R' ..." "thinking": "First, the question is: \"how many r in the word strawberry?\" I need to count the number of times the letter 'r' appears in the word \"strawberry\". Let me write down the word:...", "done_reason":"stop", "done":true, "total_duration":47975065417, "load_duration":29758167, "prompt_eval_count":10, "prompt_eval_duration":174191542, "eval_count":2514, "eval_duration":47770692833 } } [source]
Current main source defines two manifest roots under the models dir: legacy 'manifests' and new 'manifests-v2' (canonical host ollama.com). Lookup tries a digest reference, then manifests-v2, then legacy candidates; removal deletes both. Older releases only use manifests/. Layer blobs are hashed with sha256 and written via temp file 'sha256-*' in blobs/. [source]
* [x] `model` * [x] `messages` * [x] Text `content` * [x] Image `content` * [x] Base64 encoded image * [ ] Image URL * [x] Array of `content` parts * [x] `frequency_penalty` * [x] `presence_penalty` * [x] `response_format` * [x] `seed` * [x] `stop` * [x] `stream` * [x] `stream_options` * [x] `include_usage` * [x] `temperature` * [x] `top_p` * [x] `max_tokens` * [x] `tools` * [x] `reasoning_effort` (model-defined names and compatibility aliases) * [x] `reasoning` * [x] `effort` (model-defined names and compatibility aliases) * [ ] `tool_choice` * [ ] `logit_bias` * [ ] `user` * [ ] `n` [source]
Support safetensors imports through the MLX create pipeline both locally and on the server, including remote upload/staging, draft layer handling, cancellation propagation, transfer limits, and shared manifest/blob writing. [source]
./llama-quantize --imatrix imatrix.gguf --tensor-type attn_v=q5_k --tensor-type ffn_down=q5_k --prune-layers 20,21,22 input-model-f32.gguf q4_k_m 8 [source]
- `total_duration`: time spent generating the response - `load_duration`: time spent in nanoseconds loading the model - `prompt_eval_count`: number of tokens in the prompt - `prompt_eval_cached_count`: number of prompt tokens read from the cache - `prompt_eval_duration`: time spent in nanoseconds evaluating uncached prompt tokens - `eval_count`: number of tokens in the response - `eval_duration`: time in nanoseconds spent generating the response - `context`: an encoding of the conversation used in this response, this can be sent in the next request to keep a conversational memory - `response`: [source]
{ "schemaVersion": 2, "mediaType": "application/vnd.docker.distribution.manifest.v2+json", "config": { "mediaType": "application/vnd.docker.container.image.v1+json", "digest": "sha256:74156d92caf6d17ac05e9ad7b3eab0a2123f6bce4150f715ebd92e0c3af6cad1", "size": 490 }, "layers": [ { "mediaType": "application/vnd.ollama.image.model", "digest": "sha256:735af2139dc652bf01112746474883d79a52fa1c19038265d363e3d42556f7a2", "size": 291545472 }, { "mediaType": "application/vnd.ollama.image.template", "digest": "sha256:4b19ac7dd2fb1ab2f2818b73454c5a9128ca39875a8fcf686a6b1c36100a0d68", "size": 476 }, { "medi [source]
The OpenAI layer launched 2024-02-08 as 'initial experimental support' for chat completions only; embeddings, function calling, vision and logprobs were listed as future work and have since been added (except logprobs). [source]
./llama-quantize --imatrix imatrix.gguf --override-kv qwen3moe.expert_used_count=int:16 --prune-layers 20,21,22 input-model-f32.gguf pruned-model-f32.gguf copy 8 [source]
The downloaded layer's sha256 does not match the digest the model manifest promised. Ollama hashes each blob after downloading it and refuses any file whose contents do not match its own filename, because the blob store is content-addressed. The full string includes both values — want is what the manifest expects, got is what your file actually hashes to. [source]
llama.cpp will convert the LLM portion of the source model, which is enough for conversational applications. If the model accepts multimodal inputs and you wish to take advantage of them, you need to create a separate GGUF file. This file is generically known as mmproj, for "multimedia projector"; however, it may contain various components such as vision or audio encoders in addition to projections. [source]
The three runs took 32.3 s, 20.5 s and 20.6 s, with a load average of 34 to 38. Q4_K_M does not put everything in 4 bits. The tool moved the output matrix and the attn_v and ffn_down tensors of 21 of the 42 layers to Q6_K, so the average lands at 4.95 bits per weight. I also produced a 2,550.66 MiB Q8_0 for comparison. [source]
Do not rm -rf the blobs directory. Blobs are shared across models — the template, params, license and config layers are tiny files that many models reference, and a large model file you already have is a multi-gigabyte download you would be throwing away for no reason. Deleting one 42GB blob costs you 42GB; deleting the directory can cost you hundreds. [source]
That true is the cache hit, and PullModel in server/images.go uses it to set skipVerify[layer.Digest], which excludes that layer from the verification loop entirely. So a corrupt file that reached its final blob name is invisible: the pull prints pulling 4824460d29f2… 100%, then verifying sha256 digest, then success, and the bad bytes stay exactly where they are. [source]
* `values` can contain booleans (`true` or `false`) for on/off controls. It can also contain model-defined strings for named levels. * `default` is used when `think` is not set. * `values: [false]` means the model does not support thinking. * If `thinking` is omitted, the model has no thinking metadata. The model might conduct thinking based on its existing behavior. [source]
One of the challenges in rewriting manifests to the old format is that the old format involves an image configuration for each layer in the manifest, but the new format only provides one image configuration. To work around this, the registry will create synthetic image configurations for all layers except the top layer. These image configurations will not result in runnable images on their own, but only serve to fill in the parent chain in a compatible way. The IDs in these synthetic configurations will be derived from hashes of their respective blobs. The registry will create these configurat [source]
gathering model components pulling llama3.2:latest pulling manifest pulling model.bin: 100% copying file adapter.bin: 100% writing manifest success [source]
if err := verifyBlob(layer.Digest); err != nil { if errors.Is(err, errDigestMismatch) { fp, err := manifest.BlobsPath(layer.Digest) ... if err := os.Remove(fp); err != nil { slog.Info(fmt.Sprintf("couldn't remove file with digest mismatch '%s': %v", fp, err)) } } return err } [source]
// tag, and protocol scheme parts. The model and digest parts are empty. [source]
The model directory should contain the Safetensors weights for a supported architecture. [source]
Compressing a model is like compressing an image, but the loss is not distributed uniformly: some layers tolerate 4 bits, others do not. That is exactly what K and IQ quantizations exploit by allocating more bits to sensitive tensors. [source]
Standard quantization maps all weights to lower precision uniformly, which can degrade quality on the layers that matter most. The imatrix (importance matrix) path addresses this by using calibration data to identify which weights have the highest activation impact, then allocating more precision to those weights during quantization. [source]
The final channel will contain the answer to your user’s request. Check out the reasoning section for more details on the chain-of-thought. [source]
For these reasons, multimodal components are usually kept in a high-quality format such as bf16 or q8. The impact on speed and memory from using a smaller quant is negligible, but overall quality could be impacted. [source]

How it works

Ollama can pull a GGUF straight from Hugging Face with `ollama run hf.co/{username}/{repository}[:{quantization}]` without writing a Modelfile; the default quant is Q4_K_M when present, the quant tag is case-insensitive, and a full GGUF filename also works as the tag. [source]
Safetensors import: Modelfile FROM points at the directory holding the safetensors weights (FROM . when the Modelfile is in that directory); run ollama create my-model from that directory, then ollama run my-model. [source]
Deprecation in 0.34.1: typical_p can no longer be set when creating new models (existing GGUF models keep it). [source]
Most likely your framework calls Ollama's OpenAI-compatible endpoint at /v1, which ignores num_ctx in the request body because it isn't on the allowlist of OpenAI parameters Ollama maps. Only the native /api/generate and /api/chat endpoints accept it. Set OLLAMA_CONTEXT_LENGTH on the server or bake num_ctx into a Modelfile, then confirm with the CONTEXT column in ollama ps. [source]
That line comes from a test with the Modelfile in OpenBMB’s Ollama guide[7], which adds a Go TEMPLATE block. The guide states that "Ollama 0.24 does not directly evaluate the GGUF-embedded Jinja chat template". In 0.34.1 that block goes unused: its template only covers completion, so Ollama picked the GGUF one and thinking kept working. According to the 0.34.1 source, setting OLLAMA_GO_TEMPLATE=1 on the server makes Ollama use the Modelfile TEMPLATE even when the GGUF carries a template. [source]
`reasoning_effort` and `reasoning.effort` control model thinking. Use `/api/show` to discover each model's supported values and default. When a model uses its metadata to resolve named levels, supported names are applied exactly and unsupported names resolve to the model default. Boolean-only models that support enabling thinking map recognized compatibility efforts to `true`, while `"none"` requests `false`. Models without metadata retain the existing compatibility aliases: `"minimal"` maps to `"low"`, and `"xhigh"` or `"ultra"` map to `"max"`. GPT-OSS keeps its existing OpenAI compatibility [source]
TEMPLATE uses Go text/template syntax; documented variables are {{ .System }}, {{ .Prompt }} and {{ .Response }}, and text after {{ .Response }} is omitted when generating. The reference page does not document .Messages, .Tools or .Think. [source]
On Linux arm64, ollama create from the safetensors folder fails in under a second, because 0.34.1 routes that import to MLX. A Modelfile with FROM ./MiniCPM5-2B produced this: [source]
If the author already publishes a GGUF, you do not need to convert anything: ollama pull hf.co/user/repository:quantization downloads it from the Hub. The Hub documentation on Ollama[10] says that, without a tag, Q4_K_M is picked when it exists. With the official MiniCPM5-2B GGUF: [source]
Digest mismatch on pull has several reported causes: proxy ignoring Range requests, missing Content-Length yielding an empty file (sha256 e3b0c442...), disk full or short write, local memory/storage faults (got digest changes each attempt), and an existing corrupt blob accepted as a cache hit by downloadBlob(). [source]
Anyone who scripted `ollama create` against a GGUF build has a step to change, because that conversion now belongs to llama.cpp tooling. In exchange the model list gets much faster: a cold `/api/tags` load falls from 3.1 seconds to 294 milliseconds, and that is the call every Ollama client makes to find out which models are installed. [source]
reasoning_effort and reasoning.effort control model thinking. Use /api/show to discover each model’s supported values and default. When a model uses its metadata to resolve named levels, supported names are applied exactly and unsupported names resolve to the model default. Boolean-only models that support enabling thinking map recognized compatibility efforts to true, while "none" requests false. Models without metadata retain the existing compatibility aliases: "minimal" maps to "low", and "xhigh" or "ultra" map to "max". GPT-OSS keeps its existing OpenAI compatibility conversion: "minimal" [source]
Yes. digest mismatch, expected "…", got "…" comes from POST /api/blobs/:digest — the import path used by ollama create when the CLI uploads a local GGUF to the server — and returns HTTP 400. The registry pull path uses want. A mismatch on the import path means the bytes changed between your file and the server's copy of it, so check the disk and memory on the machine doing the copy, not your network. [source]
Ollama's guide advises non-quantized (non-QLoRA) adapters, because most frameworks use different quantization methods. [source]
messages = [{'role': 'user', 'content': 'What is (11434+12341)*412?'}] while True: response: ChatResponse = chat( model='qwen3', messages=messages, tools=[add, multiply], think=True, ) messages.append(response.message) print("Thinking: ", response.message.thinking) print("Content: ", response.message.content) if response.message.tool_calls: for tc in response.message.tool_calls: if tc.function.name in available_functions: print(f"Calling {tc.function.name} with arguments {tc.function.arguments}") result = available_functions[tc.function.name](**tc.function.arguments) print(f"Result: {result}") [source]
The `think` request field is supported on both /api/chat and /api/generate; when enabled, output separates reasoning from the answer, and `think: false` makes the model skip thinking and output content directly. [source]
This tool takes a GGUF input model file, typically in a high-precision format like F32 or BF16, and converts it to a quantized format. Quantization reduces the precision of model weights (e.g., from 32-bit floats to 4-bit integers), which shrinks the model's size and can speed up inference. This process however, may introduce some accuracy loss which is usually measured in Perplexity (ppl) and/or Kullback–Leibler Divergence (kld). This can be minimized by using a suitable imatrix file. [source]
The labels in the Ollama library indicate the format and size of each variant; when you omit the suffix, Ollama applies the model's default label. On Hugging Face, GGUF files include the quantization in their name. [source]
Do we want to include something here saying quantization only works when importing safetensors into MLX? [source]
- `role`: the role of the message, either `system`, `user`, `assistant`, or `tool` - `content`: the content of the message - `thinking`: (for thinking models) the model's thinking process - `images` (optional): a list of images to include in the message (for multimodal models such as `llava`) - `tool_calls` (optional): a list of tools in JSON that the model wants to use - `tool_name` (optional): add the name of the tool that was executed to inform the model of the result [source]
Disable streaming when a single JSON response is easier to process. $ curl -s http://localhost:11434/api/generate -d '{ "model": "gpt-oss:20b", "prompt": "Return OK.", "stream": false }' {"response":"OK","done":true} [source]
The useful result concerns mathematical reasoning: on GSM8K, the score drops from 77.63 in FP16 to 68.31 in Q3_K_S. The authors recommend avoiding aggressive 3-bit quantization and preferring Q4_K_S or Q5_0 when the workload involves multistep reasoning. The tasks penalized first by quantization are therefore computation, coding, and long chains of reasoning—not everyday conversation. [source]
The old path did not keep that template. The GGUF that Ollama 0.34.0 generated had 32 metadata keys and no tokenizer.chat_template, and ollama show only credited it with completion. The one llama.cpp writes has 69 keys, template included. Add TEMPLATE only when the GGUF carries no template, or when you want to force a format with OLLAMA_GO_TEMPLATE. [source]
Used when a calibration.txt is present and the target quant is one of q2_k, q3_k_m, q4_0, q4_k_m, q5_0, q5_k_m: [source]
Chat completions does not support tool_choice, logit_bias, user or n per Ollama docs. A llama-stack project fix (PR #2047, Apr 2025) worked around Ollama still calling tools when tool_choice="none" by stripping tools from the request. [source]
for await (const chunk of stream) { if (chunk.message.thinking) { thinking += chunk.message.thinking process.stdout.write(chunk.message.thinking) } if (chunk.message.content) { if (!doneThinking) { doneThinking = true process.stdout.write('\n') } content += chunk.message.content process.stdout.write(chunk.message.content) } if (chunk.message.tool_calls?.length) { toolCalls.push(...chunk.message.tool_calls) console.log(chunk.message.tool_calls) } } [source]
<Note> It is necessary to accumulate the partial fields in order to maintain the history of the conversation. This is particularly important for tool calling where the thinking, tool call from the model, and the executed tool result must be passed back to the model in the next request. </Note> [source]
Most agent frameworks talk to Ollama over the OpenAI-compatible endpoint, because that's the thing every SDK already speaks. So you set num_ctx in your framework config, the framework passes it into an OpenAI-shaped request, and Ollama drops it on the floor. Your window stays at 4096 and nothing anywhere reports a problem. [source]
When using the API, specify the `num_ctx` parameter: [source]
| Parameter | Description | Value Type | Example Usage | | - | - | - | - | | num\_ctx | Sets the size of the context window used to generate the next token. (Default: 2048) | int | num\_ctx 4096 | | repeat\_last\_n | Sets how far back for the model to look back to prevent repetition. (Default: 64, 0 = disabled, -1 = num\_ctx) | int | repeat\_last\_n 64 | | repeat\_penalty | Sets how strongly to penalize repetitions. A higher value (e.g., 1.5) will penalize repetitions more strongly, while a lower value (e.g., 0.9) will be more lenient. (Default: 1.0, disabled) | float | repeat\_penalty 1.1 | | [source]
q4_k_m — best cause score among GGUF variants, smallest size [source]
By default, the `Q4_K_M` quantization scheme is used, when it's present inside the model repo. If not, we default to picking one reasonable quant type present inside the repo. [source]
| Variable | Description | | - | - | | `{{ .System }}` | The system message used to specify custom behavior. | | `{{ .Prompt }}` | The user prompt message. | | `{{ .Response }}` | The response from the model. When generating a response, text after this variable is omitted. | [source]
When to download the GGUF with ollama pull hf.co [source]
LoRA adapters are no longer supported: merge the adapter into the base model before converting, because ADAPTER is gone [source]
When streaming, gather every chunk of `thinking`, `content`, and `tool_calls`, then return those fields together with any tool results in the follow-up request. [source]
if (chunk.message.thinking) { process.stdout.write(chunk.message.thinking) } else if (chunk.message.content) { if (inThinking) { process.stdout.write('\n\nAnswer:\n') inThinking = false } process.stdout.write(chunk.message.content) } } } [source]
Keep the MLX path focused on supported safetensors model creation with validation before MLX work, and expose that flow without the --experimental CLI gate. [source]
There is a new think parameter that can be set to true or false for enabling a model’s thinking process. When the think parameter is set to true, the output will separate the model’s thinking from the model’s output. This can help users craft new application experiences like animating the thinking process via a graphical interface, or for NPCs in games to have a thinking bubble before the output. When the think parameter is set to false, the model will not think and directly output the content. [source]
The same /v1 path has a second cost that looks unrelated: several users report the same local model returning corrupted streamed tool_calls over /v1 and correct ones over native /api/chat. If your agent forgets its rules and its tool calls come back mangled, that's one root cause, not two. And if tool calls never fire at all, that's a different Ollama issue: see Ollama does not support tools. [source]
The `/api/tags` endpoint, which Ollama clients call to list installed models, drops from a 3.1 second cold load to 294 milliseconds in version 0.34.1. That is roughly a tenfold improvement on the first call after start-up, which is the case users notice because it is the one that runs before anything is cached. [source]
In streaming, thinking chunks arrive before answer chunks; a client detects the first `thinking` chunk to open a thinking section and switches to the answer when `message.content` arrives. [source]
stopSets the stop sequences to use. When this pattern is encountered the LLM will stop generating text and return. Multiple stop patterns may be set by specifying multiple separate stop parameters in a modelfile.stringstop “AI assistant:“ [source]
Your agent ignores soul.md or drops skills on Ollama. The cause is the context window default, plus one endpoint that ignores num_ctx entirely. Hermes included. [source]
Deleting the blob is the mechanical fix. It is not the whole answer, because something produced those wrong bytes and it will do it again. The rest of this page is the diagnosis: one question — does the got digest change between attempts? — splits every report on the tracker into two groups with completely different causes, and the rest follows from that. [source]
The proposed fix, #17537 ("server: verify blob digest before trusting a same-size cache hit"), makes both pull paths hash a candidate blob before trusting it. It states the root cause in one line: both paths "use a filename-as-hash content-addressed blob store, but neither one actually confirmed the filename matched the content before skipping a re-download." It was open and unmerged when this was written. Until it lands, deleting the file yourself is not a workaround — it is the supported way to invalidate a cache entry. [source]
The PR is explicit about why this is worth naming separately: "The empty blob is caught later by verifyBlob, so nothing corrupt gets stored, but the user sees a digest mismatch. That reads like a corrupted or tampered blob when the actual cause is a registry that did not report a size, which sends you looking in entirely the wrong place." [source]
No. It only disables the startup sweep that removes partial downloads and unreferenced blobs, and it has no effect on digest verification. It is useful when you want a half-finished download or a blob you are inspecting to survive a server restart, not when you want a corrupt one repaired. [source]
On Ollama 0.12+, thinking-capable models (qwen3 thinking variants, deepseek-r1, deepseek-v3.1) auto-enable thinking when `think` is unset; standard models such as qwen2.5 and llama3.2 do not. [source]
Alternatively, you can change the amount of time all models are loaded into memory by setting the `OLLAMA_KEEP_ALIVE` environment variable when starting the Ollama server. The `OLLAMA_KEEP_ALIVE` variable uses the same parameter types as the `keep_alive` parameter types mentioned above. Refer to the section explaining [how to configure the Ollama server](#how-do-i-configure-ollama-server) to correctly set the environment variable. [source]
When thinking is enabled, the output will separate the model’s thinking from the model’s output. When thinking is disabled, the model will not think and directly output the content. [source]
When deploying Qwen3, Olama can support the enable_thinking parameter. The official documentation has stated that enable_thinking=False can support disabling thinking, but after passing the parameter in Olama deployment, it still enters thinking mode. Only by adding /no_think during each inference can we ensure that no thinking is done. Can additional general parameters be added in Olama and passed transparently to the original model to facilitate the specific characteristics of the model [source]
for await (const chunk of response) { if (chunk.message.thinking && !startedThinking) { startedThinking = true process.stdout.write('Thinking:\n========\n\n') } else if (chunk.message.content && startedThinking && !finishedThinking) { finishedThinking = true process.stdout.write('\n\nResponse:\n========\n\n') } [source]
Server-wide, OLLAMA_CONTEXT_LENGTH. The default for every request that doesn't carry its own. This is the one to use for agents, because it survives the OpenAI-compatible endpoint problem above. [source]
There are dangerous arbitrary code execution attacks that can be perpetrated when you load a pickle file. We suggest loading models from users and organizations you trust, relying on signed commits, and/or loading models from TF or Jax formats with the `from_tf=True` auto-conversion mechanism. We also alleviate this issue by displaying/"vetting" the list of imports in any pickled file, directly on the Hub. Finally, we are experimenting with a new, simple serialization format for weights called [`safetensors`](https://github.com/huggingface/safetensors). [source]
--leave-output-tensor leave output.weight un(re)quantized. Increases model size but may also increase quality, especially when requantizing [source]
Why the num_ctx you already set did nothingWhat Ollama actually does when the prompt is too longThe token math that makes this clickCheck what your model actually receivedThree ways to set it, and which to useDon't trust any published "default" for your model, including this pageHow much headroom, and what it costsHermes, OpenClaw, and the framework layerThe fix you should make even after fixing num_ctxFrequently Asked Questions [source]
would it be possible to add an env option so that num_ctx from the request is ignored and always set to the value of OLLAMA_CONTEXT_LENGTH? [source]
The rationale behind this is that for a shared Ollama instance, it's unfortunate if users cause a runner restart by setting an incorrect num_ctx value (even one smaller than the current value). [source]
Point the models directory at a different disk and pull there. OLLAMA_MODELS=/some/other/disk/models on the server process — not your shell — and re-run. If it verifies on one disk and not another, you are done diagnosing. [source]
The nominal number of bits per weight. The actual value is higher because sensitive tensors remain at higher precision: Q4_K_M measures 4.89 bits per weight. [source]
* <strong>auto-enable thinking by default</strong> when this field is not set.</li> [source]
* process in the response's thinking field. [source]

Parameters and configuration

gpt-oss uses named effort levels instead of booleans: /api/show example shows values ["low","medium","high"] with default "medium"; Spring AI documents that gpt-oss takes string levels and not boolean enable/disable. [source]
gpt-oss natively takes reasoning effort low, medium or high set in the system message (e.g. `Reasoning: high`), defaults to medium, and writes raw chain of thought on the `analysis` channel, which OpenAI says should not be shown to end users; this is why Ollama exposes gpt-oss as string levels rather than a boolean. [source]

How-to and procedures

To import a Hugging Face model into Ollama 0.34.1, download the safetensors with hf download, convert them with convert_hf_to_gguf.py, quantize with llama-quantize and create the model with a Modelfile that sets num_ctx and, on a shared machine, num_thread. Do not write TEMPLATE if the GGUF carries its own template. On 15 September there was already a v0.34.2-rc0 pre-release whose notes only say "llama.cpp updates"; I checked them on 16 September and they do not change this procedure. Before repeating this guide on another version, read its notes on the Ollama releases page[11]. [source]
To import a Safetensors LoRA adapter, write a Modelfile with FROM <base model name> and ADAPTER /path/to/safetensors/adapter/directory, then run ollama create. ADAPTER . works when the adapter sits beside the Modelfile. [source]
Use `/api/show` to discover a model's supported thinking values and default. `thinking.type: "enabled"` and `"disabled"` act as explicit on/off controls. When a model uses its metadata to resolve named levels, supported `output_config.effort` names are applied exactly and unsupported names resolve to the model default. Models without metadata keep the existing compatibility behavior, which trims and lowercases effort names and maps `"xhigh"` to `"high"`. [source]
To change sampling parameters, create a file named `params` in the repository. The file must be in JSON format. For the list of all available parameters, please refer to [this documentation](https://github.com/ollama/ollama/blob/main/docs/modelfile.mdx#parameter). [source]
To import a GGUF model, create a `Modelfile` containing: [source]
Create a model from GGUF files. The `files` parameter should contain the file name and SHA256 digest of each GGUF file. For a split GGUF model, include each shard under its original split filename. Use [/api/blobs/:digest](#push-a-blob) to push each file before calling this API. [source]
Use the API key in the `Authorization: Bearer` header. This also applies to the hosted Anthropic-compatible `/v1/messages` endpoint; `x-api-key` alone is not supported. [source]
Use GGUF community tools to prepare and quantize GGUF files before importing. [source]
To know more about the Go template format, please refer to [this documentation](https://github.com/ollama/ollama/blob/main/docs/template.mdx) [source]
Use `ollama create` to then create the quantized model. [source]
To invoke Ollama’s OpenAI compatible API endpoint, use the same OpenAI format and change the hostname to http://localhost:11434: [source]
Use /api/show to discover a model’s supported thinking values and default. thinking.type: "enabled" and "disabled" act as explicit on/off controls. When a model uses its metadata to resolve named levels, supported output_config.effort names are applied exactly and unsupported names resolve to the model default. Models without metadata keep the existing compatibility behavior, which trims and lowercases effort names and maps "xhigh" to "high". [source]
Set the `think` field on a chat or generate request: [source]
To calculate how fast the response is generated in tokens per second (token/s), divide `eval_count` / `eval_duration` \* `10^9`. [source]
Create safetensors models through a hybrid local/remote pipeline. [source]
To view the Modelfile of a given model, use the `ollama show --modelfile` command. [source]
Enable experimental safetensors model creation from local weights [source]
Use the `ollama create mymodel` command to create a new model with the updated context size. Call the API with the updated model name: [source]
Use this capability to audit model steps, animate the model *thinking* in a UI, or hide the trace entirely when you only need the final response. [source]
Use this tool to execute Python code in your chain of thought. The code will not be shown to the user. This tool should be used for internal reasoning, but not for code that is intended to be visible to the user (e.g. when creating plots, tables, or files). [source]
To make Gemma 3 even more accessible, we are announcing new versions optimized with Quantization-Aware Training (QAT) that dramatically reduces memory requirements while maintaining high quality. This enables you to run powerful models like Gemma 3 27B locally on consumer-grade GPUs like the NVIDIA RTX 3090. [source]

Examples and snippets

```json5 — `modelfile: '# Modelfile generated by "ollama show"\n# To build a new Modelfile based on this one, replace the ` … [source]
```json — `"model": "llama3.2",` … [source]
```sh — `ollama run hf.co/bartowski/Llama-3.2-3B-Instruct-GGUF:IQ3_M` … [source]
```shell theme={"system"} — `curl http://localhost:11434/api/generate -d '{` … [source]
``` — `FROM llama3.2` … [source]
```shell — `curl http://localhost:11434/api/chat -d '{` … [source]
``` — `# Modelfile generated by "ollama show"` … [source]
```shell theme={"system"} — `curl https://ollama.com/api/chat \` … [source]
```shell theme={"system"} — `curl http://localhost:11434/api/generate -d '{"model": "llama3.2", "keep_alive": -1}'` … [source]
```shell theme={"system"} — `curl http://localhost:11434/api/generate -d '{"model": "llama3.2", "keep_alive": 0}'` … [source]
```json theme={"system"} — `"model": "gemma4",` … [source]
```shell — `curl http://localhost:11434/api/generate -d '{` … [source]
```json — `"model": "llama3.1",` … [source]
```python theme={"system"} — `from ollama import chat, web_fetch, web_search` … [source]
```shell — `ollama create my-model` … [source]
``` — `usage: llama-export-lora [options]` … [source]
```shell — `curl http://localhost:11434/api/create -d '{` … [source]
```shell theme={"system"} — `curl http://localhost:11434/api/show -d '{` … [source]
```json5 — `"model": "codellama:code",` … [source]
```json — `"model": "llama3.1:8b",` … [source]
```shell — `curl -T model.gguf -X POST http://localhost:11434/api/blobs/sha256:29fdb92e57cf0827ded04ae6461b5931d01fa595843` … [source]
```shell theme={"system"} — `ollama create my-model` … [source]
```json — `"models": [` … [source]
```shell — `curl -X POST http://localhost:11434/api/chat -H "Content-Type: application/json" -d '{` … [source]
```shell theme={"system"} — `curl http://localhost:11434/v1/chat/completions \` … [source]
```shell theme={"system"} — `curl http://localhost:11434/v1/chat/completions \` … [source]
```powershell theme={"system"} — `(Invoke-WebRequest -method POST -Body '{"model":"llama3.2", "prompt":"Why is the sky blue?", "stream": false}'` … [source]
```json theme={"system"} — `"$schema": "https://opencode.ai/config.json",` … [source]
```shell theme={"system"} — `/set parameter num_ctx 4096` … [source]
```shell — `$ ollama create --quantize q4_K_M mymodel` … [source]
```shell theme={"system"} — `curl http://localhost:11434/api/generate -d '{"model": "mistral"}'` … [source]
```shell — `curl -X POST http://localhost:11434/api/generate -H "Content-Type: application/json" -d '{` … [source]
```json — `"model": "llava",` … [source]
```dockerfile — `FROM /path/to/file.gguf` … [source]
```shell — `{"status":"importing fred (2 tensors)"}` … [source]
``` — `{{ if .System }}<|system|>` … [source]
```dockerfile — `FROM <base model name>` … [source]
```dockerfile — `FROM <model name>` … [source]
```json — `"model": "mistral",` … [source]
```shell — `curl http://localhost:11434/api/copy -d '{` … [source]
```shell — `curl -X DELETE http://localhost:11434/api/delete -d '{` … [source]
```shell — `curl http://localhost:11434/api/push -d '{` … [source]
```shell — `curl http://localhost:11434/api/ps` … [source]
```shell — `curl http://localhost:11434/api/version` … [source]
```sh — `ollama run hf.co/{username}/{repository}:{quantization}` … [source]
```toml theme={"system"} — `model = "gpt-oss:120b"` … [source]
```shell theme={"system"} — `ollama run my-model` … [source]
```json — `{"status":"parsing GGUF"}` … [source]
```shell — `curl -I http://localhost:11434/api/blobs/sha256:29fdb92e57cf0827ded04ae6461b5931d01fa595843f55d36f5b275a52087d` … [source]
```dockerfile theme={"system"} — `FROM /path/to/safetensors/directory` … [source]
```dockerfile theme={"system"} — `FROM /path/to/file.gguf` … [source]
```dockerfile theme={"system"} — `FROM /path/to/model-*.gguf` … [source]
``` — `Thinking: Okay, the user is asking about Ollama's new engine. I need to figure out what they're referring to.` … [source]
```shell — `HEAD /api/blobs/:digest` … [source]
``` — `POST /api/blobs/:digest` … [source]
```shell theme={"system"} — `docker build -t ollama-with-ca .` … [source]
```nginx theme={"system"} — `server {` … [source]
```shell theme={"system"} — `ngrok http 11434 --host-header="localhost:11434"` … [source]
```shell theme={"system"} — `cloudflared tunnel --url http://localhost:11434 --http-host-header="localhost:11434"` … [source]
```json — `"status": "verifying sha256 digest"` … [source]
```json theme={"system"} — `{"model":"gemma4","created_at":"2025-10-26T17:15:24.097767Z","response":"That","done":false}` … [source]
```shell theme={"system"} — `export ANTHROPIC_AUTH_TOKEN=ollama # required but ignored` … [source]
``` — `FROM <some model>` … [source]
```shell choice.sh theme={"system"} — `curl http://localhost:11434/v1/systemone \` … [source]
```shell image.sh theme={"system"} — `curl http://localhost:11434/v1/systemone \` … [source]
```shell multiple.sh theme={"system"} — `curl http://localhost:11434/v1/systemone \` … [source]
```python theme={"system"} — `from ollama import chat` … [source]
```shell theme={"system"} — `docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama` … [source]
```shell theme={"system"} — `docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/` … [source]
```shell theme={"system"} — `docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/` … [source]
```shell theme={"system"} — `ANTHROPIC_AUTH_TOKEN=ollama ANTHROPIC_BASE_URL=http://localhost:11434 ANTHROPIC_API_KEY="" claude --model kimi` … [source]
```shell theme={"system"} — `export COPILOT_PROVIDER_BASE_URL=http://localhost:11434/v1` … [source]
```shell theme={"system"} — `COPILOT_PROVIDER_BASE_URL=http://localhost:11434/v1 COPILOT_PROVIDER_API_KEY= COPILOT_PROVIDER_WIRE_API=respon` … [source]
```json theme={"system"} — `"custom_models": [` … [source]
```json theme={"system"} — `"custom_models": [` … [source]
```shell theme={"system"} — `export POOLSIDE_STANDALONE_BASE_URL=http://localhost:11434/v1` … [source]
```shell theme={"system"} — `POOLSIDE_STANDALONE_BASE_URL=http://localhost:11434/v1 POOLSIDE_API_KEY=ollama pool -m kimi-k2.6:cloud` … [source]
```json theme={"system"} — `"thinking": {` … [source]
``` — `FROM ./ollama-model.gguf` … [source]
``` — `FROM ./model-*.gguf` … [source]
```bash — `./bin/llama-export-lora \` … [source]
```dockerfile — `FROM /path/to/safetensors/directory` … [source]
```python theme={"system"} — `from ollama import chat` … [source]
```shell — `curl http://localhost:11434/api/tags` … [source]
```shell — `curl http://localhost:11434/api/show -d '{` … [source]
```shell — `curl http://localhost:11434/api/pull -d '{` … [source]
```shell — `curl http://localhost:11434/api/embed -d '{` … [source]
```json — `"model": "all-minilm",` … [source]
```shell — `curl http://localhost:11434/api/embed -d '{` … [source]
```shell — `curl http://localhost:11434/api/embed -d '{` … [source]
```shell — `curl http://localhost:11434/api/embed -d '{` … [source]
```shell — `curl http://localhost:11434/api/embeddings -d '{` … [source]

Measurements and reference values

Since Ollama 0.34.1, ollama create no longer converts or quantizes safetensors weights. To import a Hugging Face model, download it with hf download, convert it to GGUF with llama.cpp's convert_hf_to_gguf.py, quantize it with llama-quantize and build it from a Modelfile. With MiniCPM5-2B on a CPU, the three stages took 35 s, 21 s and 2 s. [source]
Ollama 0.34.1 makes ollama create from MLX safetensors a stable feature. Building a GGUF model now requires llama.cpp tooling for safetensor conversion and quantization, and the /api/tags cold load falls from 3.1 seconds to 294 ms. [source]
ollama create -q q4_K_M from a BF16 GGUF Quantizes: 1.6 GB in 17 s create-time quantization is only supported for safetensors imports [source]
llama3.1 library tags follow <size>-<instruct|text>-<quant>, with suffixes q2_K, q3_K_S/M/L, q4_0, q4_1, q4_K_S/M, q5_0, q5_1, q5_K_S/M, q6_K, q8_0 and fp16; the page lists 93 tags, sizes from 3.2GB (8b q2_K) to 812GB (405b fp16). [source]
Disk use adds up to 5.03 GB of safetensors, 5.04 GB for the BF16 GGUF, 1.56 GB for the quantized GGUF and 1.56 GB more for Ollama’s copy. [source]
On this CPU, Q8_0 processed the prompt 58% faster than Q4_K_M and generated at the same pace, and Q4_K_M raises perplexity by 5.5% over Q8_0. Q4_K_M takes 42% less disk than Q8_0; in memory, /api/ps reported 2.03 GB with num_ctx 8192 and 1.79 GB with the default context. BF16 is only an intermediate step: at 2.28 tokens/s it is not usable on this processor. [source]
llama3.1:latest, llama3.1 and llama3.1:8b are the same model (4.9GB, ID 46e0c10c039e). The page does not name its quantization, but the ID matches llama3.1:8b-instruct-q4_K_M, so the default is most likely Q4_K_M. gemma3:latest (3.4GB) likewise shares a digest with gemma3:4b-it-q4_K_M. [source]
ollama create -q q4_K_M from safetensors Converts and quantizes: 1.6 GB in 106 s unsupported --quantize "q4_K_M" [source]
The size of a GGUF file follows this formula: parameters in billions multiplied by bits per weight, divided by eight, gives gigabytes. For a 70B in Q4_K_M: 70 × 4.89 ÷ 8 ≈ 42.8 GB, as confirmed by the llama.cpp table for Llama 3.1 70B (43.1 GB). Then add the KV cache, which grows with the context, plus a margin for the system. [source]
The full Docker image: ghcr.io/ggml-org/llama.cpp:full-v0.4.1 takes 3.91 GB and ships convert_hf_to_gguf.py, llama-quantize and the Python dependencies [source]
```shell vision.sh theme={"system"} curl -X POST http://localhost:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "qwen3-vl:8b", "messages": [{ "role": "user", "content": [{"type": "text", "text": "What is this an image of?"}, {"type": "image_url", "image_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAG0AAABmCAYAAADBPx+VAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAA3VSURBVHgB7Z27r0zdG8fX743i1bi1ikMoFMQloXRpKFFIqI7LH4BEQ+NWIkjQuSWCRIEoULk0gsK1kCBI0IhrQVT7tz/7zZo888yz1r7MnDl7z5xvsjkzs2fP3uu71nNfa7lkAsm7d++Sffv2JbNmzUqcc8m0adOSzZs3Z+ [source]
Two practical consequences. First, switching from Q4_K_M to Q8_0 for an imperceptible quality gain costs about 30% of generation speed in these measurements. Second, weight quantization and KV-cache quantization are two separate settings: the latter reduces long-context memory usage and is covered in its own guide. [source]
With MiniCPM5-2B, conversion to BF16 took 35 s, quantization to Q4_K_M 21 s and the import 2 s, medians of three runs. [source]
The 1.6 GB download took 21 s, and the model showed the same capabilities as the hand-imported one. The Hub also added five stop sequences, because the repository has no params file: <s>, <|im_start|>, <|im_end|>, <think> and <|im_start|>user. Import the model yourself when no GGUF exists, when you want a quantization the author does not publish, or when you start from your own fine-tune. To pick a size and variant, compare with Gemma 4 locally with Ollama. [source]
NF4/GPU is the only variant competitive on complex incidents (20% win rate). All GGUF quantized variants collapse to 0% wins on complex incidents — quantization significantly degrades performance on long, evidence-heavy DAGs. [source]
stream = chat( model='qwen3', messages=[{'role': 'user', 'content': 'What is 17 × 23?'}], think=True, stream=True, ) [source]
async function main() { const stream = await ollama.chat({ model: 'qwen3', messages: [{ role: 'user', content: 'What is 17 × 23?' }], think: true, stream: true, }) [source]
curl -X POST http://localhost:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "qwen3-vl:8b", "messages": [{ "role": "user", "content": [{"type": "text", "text": "What is this an image of?"}, {"type": "image_url", "image_url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAG0AAABmCAYAAADBPx+VAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAA3VSURBVHgB7Z27r0zdG8fX743i1bi1ikMoFMQloXRpKFFIqI7LH4BEQ+NWIkjQuSWCRIEoULk0gsK1kCBI0IhrQVT7tz/7zZo888yz1r7MnDl7z5xvsjkzs2fP3uu71nNfa7lkAsm7d++Sffv2JbNmzUqcc8m0adOSzZs3Z+/XES4ZckAWJEGWPiCxjsQNLWmQsWjRIpMsea [source]
You need Ollama 0.34.1, llama.cpp’s conversion tools and the Hugging Face command line, plus about 13 GB free for a 2.5-billion-parameter model. If you do not have Ollama yet, follow the guide to installing Ollama on your computer. These are the pieces; you need only one of the first two, because both produce the same file: [source]
12 GB8B in Q6_K or Q8_0, or 14B in Q4_K_M (8.6 GB) [source]
q8_0 — 32.6% win rate, nearly identical avg score to fp16 [source]
Without Docker, clone the v0.4.1 tag and install the converter requirements in a virtual environment. The install took 24 s and left torch 2.11.0 for CPU, transformers 4.57.6 and gguf 0.19.0: [source]
response = client.chat.completions.create( model='qwen3-vl:8b', messages=[ { 'role': 'user', 'content': [ {'type': 'text', 'text': "What's in this image?"}, { 'type': 'image_url', 'image_url': 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAG0AAABmCAYAAADBPx+VAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAA3VSURBVHgB7Z27r0zdG8fX743i1bi1ikMoFMQloXRpKFFIqI7LH4BEQ+NWIkjQuSWCRIEoULk0gsK1kCBI0IhrQVT7tz/7zZo888yz1r7MnDl7z5xvsjkzs2fP3uu71nNfa7lkAsm7d++Sffv2JbNmzUqcc8m0adOSzZs3Z+/XES4ZckAWJEGWPiCxjsQNLWmQsWjRIpMseaxcuTKpG/7HP27I8P79e7dq1ars/yL4/v27S0ejqwv+cUOGEGGpKHR37tzJCEpHV9tn [source]
const response = await openai.chat.completions.create({ model: "qwen3-vl:8b", messages: [ { role: "user", content: [ { type: "text", text: "What's in this image?" }, { type: "image_url", image_url: "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAG0AAABmCAYAAADBPx+VAAAACXBIWXMAAAsTAAALEwEAmpwYAAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAA3VSURBVHgB7Z27r0zdG8fX743i1bi1ikMoFMQloXRpKFFIqI7LH4BEQ+NWIkjQuSWCRIEoULk0gsK1kCBI0IhrQVT7tz/7zZo888yz1r7MnDl7z5xvsjkzs2fP3uu71nNfa7lkAsm7d++Sffv2JbNmzUqcc8m0adOSzZs3Z+/XES4ZckAWJEGWPiCxjsQNLWmQsWjRIpMseaxcuTKpG/7HP27I8P79e7dq1ars/yL4/v27S0ejqwv+cUOGEGGpKHR37tzJCEp [source]
A 32B in Q4_K_M (19.6 GB) doesn't fit comfortably on 24 GB with a long context: the KV cache takes several GB, plus compute buffers. To quantify it, use the VRAM calculator or read the guide to the context window. [source]
┌─────────┬────────┬────────┐ │ GGUF │ Local │ Remote │ ├─────────┼────────┼────────┤ │ 12.1 GB │ 9.5 s │ 11.3 s │ ├─────────┼────────┼────────┤ │ 29.6 GB │ 22.1 s │ 26.4 s │ └─────────┴────────┴────────┘ [source]
Only for gigantic models, when nothing else fits. With this kind of quantization, fidelity can be measured: see the Kimi K3 example, where dynamic 1-bit achieves only 78.9% agreement with the original. [source]
The download fetched 11 files and 5.05 GB in 68 s, including the package install. The folder holds config.json, tokenizer.json, chat_template.jinja and a single 5,033,557,096-byte model-00000-of-00001.safetensors. Without a token, hf warns that anonymous requests get lower limits; for a gated model, export HF_TOKEN first. [source]
llama_model_quantize_impl: model size = 4800.66 MiB (16.00 BPW) llama_model_quantize_impl: quant size = 1484.08 MiB (4.95 BPW) llama_quantize: quantize time = 20647.62 ms [source]
The import took 2.25 s, 1.93 s and 1.99 s over three runs with a load average of 5.5. The ollama show command confirmed the architecture and the capabilities Ollama detected in the GGUF: [source]
Un Modelfile de Ollama es una configuración que permite personalizar y gestionar las modelos que utiliza para la creación de contenido. total duration: 606.975347ms prompt eval count: 41 token(s) prompt eval cached: 40 token(s) eval count: 30 token(s) eval rate: 52.48 tokens/s [source]
/api/tags is much faster on large model libraries (3.1 s → 294 ms cold in testing), and model capabilities are now reported consistently. [source]
Exemple : 32 milliards en Q5_K_M 32 × 5,70 ÷ 8 ≈ 22,8 Go + cache KV (selon le contexte) + 1 à 2 Go de marge [source]
Similar in size to Q4_K_M, minus ~9%. Useful when VRAM is tight. [source]
A 13B at about 5 GB in size, with a noticeable loss in reasoning. [source]
6 to 8 GB7-8B in Q4_K_M; IQ4_XS if it doesn’t fit [source]
16 GB14B in Q5_K_M (10 GB) with a comfortable context [source]
24 GB32B in Q4_K_M (19.6 GB) with a moderate context [source]
32 GB32B in Q5_K_M (22.8 GB) or Q6_K [source]
64 GB and more70B in Q4_K_M (42.8 GB) [source]
Best local LLM for coding in 2026Best local LLM in French in 2026Best local LLM for RAG in 2026Best local LLM for reasoning in 2026Best local LLM with vision in 2026Best local LLM for agents / tool use in 2026Best LLM on RTX 4090 (24 GB) in 2026Best LLM for 16 GB of VRAM in 2026 + 64 more rankings Best LLM on RTX 3060 (12 GB) in 2026Best LLM for 8 GB of VRAM in 2026Best LLM on Mac Apple Silicon in 2026Best LLM without a GPU (CPU only) in 2026Best open local LLM for commercial useBest sovereign European LLM in 2026Best LLM for MacBook Air M4 (M4) in 2026Best LLM for MacBook Air M3 in 2026Best L [source]
q8_0 — near-NF4 quality on simple/medium, ~2× memory saving [source]
q4_k_m — 986 MB, competitive on both tasks [source]
Q4_K_M 1.56 GB 104.68 tokens/s 32.52 tokens/s 26.90 ± 1.18 [source]
Q8_0 2.68 GB 165.60 tokens/s 33.70 tokens/s 25.50 ± 1.12 [source]
BF16 5.04 GB 2.67 tokens/s 2.28 tokens/s Not measured [source]
Gemma 3 27B: Drops from 54 GB (BF16) to just 14.1 GB (int4) [source]
Gemma 3 12B: Shrinks from 24 GB (BF16) to only 6.6 GB (int4) [source]
Gemma 3 4B: Reduces from 8 GB (BF16) to a lean 2.6 GB (int4) [source]
Gemma 3 1B: Goes from 2 GB (BF16) down to a tiny 0.5 GB (int4) [source]

Problems, failure modes and limitations

Since Ollama 0.34.1, ollama create no longer converts or quantizes the weights you download from Hugging Face: llama.cpp does that part now. The release is dated 14 September 2026, and the one-line command that turned a safetensors folder into a GGUF now answers with an error. This guide imports MiniCPM5-2B end to end with the tools that worked on 16 September, with the times, sizes and errors I got on an arm64 CPU with no GPU. There is also a Spanish version of this guide. [source]
On Linux arm64 (blog test), safetensors import fails with 'MLX runtime is not available: failed to load MLX dynamic library'; with --force the import succeeds but ollama run returns 'Error: 500 Internal Server Error: MLX runtime is not available'. The author attributes this to the MLX build shipping only in the amd64 image and did not test amd64 or Apple silicon. [source]
Anthropic layer supports messages, streaming, system prompts, tools and tool results, base64 images, thinking blocks and output_config.effort. Not supported: /v1/messages/count_tokens, tool_choice, metadata, cache_control prompt caching, Batches, citations, PDF document blocks, streaming error events. budget_tokens is accepted but not enforced; token counts are approximate. [source]
Per the api.md reference, done_reason "load" is returned for a chat call with empty messages (loads the model) and "unload" for an empty prompt/messages with keep_alive:0. keep_alive defaults to 5m. Raw mode (raw:true) skips templating and returns no `context`; `context` is deprecated. [source]
Issue #13775 is the instructive one here, partly because of how it ended. The reporter had a 6.7GB GGUF, a sha256sum on disk that matched Hugging Face, and ollama create failing with a different "got" hash on each attempt. A maintainer downloaded the same file, hashed it, hashed the resulting blob, and got the expected digest on both — then asked the reporter to check for memory or disk read errors. The reporter closed it themselves with "i genuinely believe it's not an entry issue with ollama. But i also can't figure out wtf it is." Same signature, same conclusion: a changing got is a local r [source]
| Feature | Description | | - | - | | `/v1/messages/count_tokens` | Token counting endpoint | | `tool_choice` | Forcing specific tool use or disabling tools | | `metadata` | Request metadata (user\_id) | | Prompt caching | `cache_control` blocks for caching prefixes | | Batches API | `/v1/messages/batches` for async batch processing | | Citations | `citations` content blocks | | PDF support | `document` content blocks with PDF files | | Server-sent errors | `error` events during streaming (errors return HTTP status) | [source]
All three quants are equivalent on risk. q4_k_m, q5_k_m, and q8_0 all win 26.9% of risk incidents. There is no meaningful quality degradation from lower quantization on the risk task. [source]
Merging a PEFT LoRA into a quantized base re-quantizes the base weight, so repeated merge/unmerge cycles accumulate rounding error; for deployment prefer one merge_and_unload() (assign the returned model) and keep the unmerged adapter, which supports merging offline before an Ollama FROM import as an alternative to ADAPTER. [source]
Limit GGUF create to wrapping existing GGUF inputs into Ollama manifests. Remove the in-tree safetensors-to-GGUF converter, server quantization path, and converter-only dependencies so GGUF conversion and quantization stay in llama.cpp tooling. [source]
PR 18448[3] explains the typical_p case: GGUF models that already carry it keep it, but you cannot create a new one with that parameter. Version 0.34.1 also brings changes unrelated to importing, such as /api/tags dropping from 3.1 s to 294 ms cold in the testing its notes cite. [source]
Errors raised before streaming begins return an HTTP status (400 bad request/invalid JSON, 404 model not found, 429 rate limit, 500 internal, 502 bad gateway when a cloud model is unreachable) with a JSON body {"error": "<message>"}. [source]
The Spanish agreement error in the answer ("las modelos") comes from MiniCPM5-2B itself, which makes mistakes in Spanish at 2.5 billion parameters. In another test with thinking on, it defined GGUF as an audio archive format. Without --think=false, MiniCPM5-2B reasons before answering and Ollama returns that text in the API’s thinking field, separate from the answer. If you plan to give it tools, the guide to function calling with Ollama shows how to declare them. [source]
- `truncate`: truncates the end of each input to fit within context length. Returns error if `false` and context length is exceeded. Defaults to `true` - `options`: additional model parameters listed in the documentation for the [Modelfile](./modelfile.mdx#valid-parameters-and-values) such as `temperature` - `keep_alive`: controls how long the model will stay loaded into memory following the request (default: `5m`) - `dimensions`: number of dimensions for the embedding [source]
If an error occurs after streaming has started, the server emits a line with an `error` property in the NDJSON stream and the HTTP status stays 200. Docs example: {"error":"an error was encountered while running the model"}. [source]
Robust client rule: parse each line separately, check for an `error` key before reading content, and treat a stream that ends without done:true as incomplete. A reader that only joins `response` fields can return partial text and silently drop the error line. [source]
Clients that expect a single JSON object from a streaming-default endpoint fail with parse errors (e.g. 'JSON Parse error: Unexpected EOF'); the workaround is stream:false. [source]
* [x] `message_start` * [x] `content_block_start` * [x] `content_block_delta` (text\_delta, input\_json\_delta, thinking\_delta) * [x] `content_block_stop` * [x] `message_delta` * [x] `message_stop` * [x] `ping` * [x] `error` [source]
The third variant, for completeness: the fast-transfer path used for safetensors models (x/transfer/download.go) returns a bare digest mismatch with no want/got at all, after removing its own .tmp file. If your error has no digests in it, that is the path you are on. [source]
Anthropic-compatible proxies can fail on Claude Code's experimental beta parameters (400 errors); agentgateway documents CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1 as the workaround. Whether this applies to Ollama is untested here. The hosted Ollama endpoint rejects anthropic-version older than 2023-06-01. [source]
importing safetensors model Error: MLX runtime is not available: failed to load MLX dynamic library (searched: [/usr/lib/ollama …]) [source]
PR #15489 chased the same pattern much further, and its findings are the single most useful thing on the tracker for this class of failure. On a machine with a reproducible mismatch, the author confirmed that the bytes on the wire were correct — an inline sha256 tee'd off the HTTP response body during download matched the expected digest, and curl piped to sha256sum against the same URL also matched. The blob was even functional: ollama run on it loaded and generated coherent text. But re-reading the file from disk with sha256sum, with io.Copy(sha256.New(), f) and with several other approaches [source]
Has this problem been resolved? Remove the issue of empty blocks in. Additionally, using/hidethinking does not work and cannot remove empty blocks. Furthermore, using the compiled version for startup also does not work with the think parameter [source]
If an error occurs mid-stream, the error will be returned as an object in the `application/x-ndjson` format with an `error` property. Since the response has already started, the status code of the response will not be changed. [source]
This fix detects failure; it does not solve the underlying cause. A timeout, proxy disconnection, model loading problem, or user cancellation may need different handling. A completion marker also does not prove factual correctness or that the model satisfied the request. Tool-call responses and OpenAI-compatible event streams need their own readers. [source]
Quantization degrades cause analysis more than risk assessment. Cause scores drop 2–4 points with quantization while risk scores remain roughly stable or improve slightly. Cause analysis requires precise evidence grounding from the DAG and is more sensitive to precision loss. [source]
/Ollama Digest Mismatch Error: Delete One Bad Blob [source]
Ollama Digest Mismatch Error: Delete One Bad Blob [source]
Delete the single blob named after the want digest, then pull again. The error prints want sha256:abc123…; the file is at <models>/blobs/sha256-abc123… — same string, colon swapped for a hyphen. That is the whole mapping, and it means you never need to wipe the blobs directory. The reason retrying alone so often changes nothing is that downloadBlob() in server/download.go treats any file already sitting at the blob path as a cache hit — it calls os.Stat and nothing else — which then sets skipVerify for that layer, so the corrupt file is neither re-hashed nor replaced. Remove the file, and the [source]
func BlobsPath(digest string) (string, error) { // only accept actual sha256 digests pattern := "^sha256[:-][0-9a-fA-F]{64}$" ... digest = strings.ReplaceAll(digest, ":", "-") path := filepath.Join(envconfig.Models(), "blobs", digest) ... } [source]
So if the mismatch happened on a fresh download, the blob is deleted for you and a retry starts clean. If you are seeing the same error over and over anyway, either the removal failed (that couldn't remove file with digest mismatch line will be in the server log — usually a permissions problem on a Linux install where the daemon runs as the ollama user) or the same wrong bytes are arriving every time. Which brings us to the diagnostic that matters. [source]
If the blob verifies by hand but run still fails, you are no longer debugging a digest mismatch — go to Ollama 500 Internal Server Error and read the server log. [source]
Ollama documents that errors during streaming arrive as NDJSON objects with an error property. Once streaming starts, the response status is not changed. That means an application which checks only HTTP 200 can miss a later failure. Official error reference [source]
The --force flag skips that validation. With it, Ollama imported the model in 26 s as 386 layers and 5.0 GB, but ollama run returned Error: 500 Internal Server Error: MLX runtime is not available. Deleting it with ollama rm shrank the blob store from 7.7 GB to 3.0 GB. [source]
func verifyBlob(digest string) error { fp, err := manifest.BlobsPath(digest) ... fileDigest, _ := GetSHA256Digest(f) if digest != fileDigest { return fmt.Errorf("%w: want %s, got %s", errDigestMismatch, digest, fileDigest) } return nil } [source]
There is one adjacent behaviour worth knowing, because it produces a confusing log line. If manifest.Manifests() cannot parse your manifests, the server logs corrupt manifests detected, skipping prune operation. Re-pull or delete to clear and skips the sweep entirely. A truncated or zero-length manifest file — the same VM-hang or disk-full scenario that damages blobs — will do that, and no amount of blob deletion fixes a broken manifest. Delete the offending file under manifests/ and pull the model again. [source]
Deprecated typical_p: it can no longer be set when creating new models, existing GGUF models retain support. [source]
1 - Error occurred (invalid Modelfile, missing base model, etc.) [source]
Error: specified Modelfile wasn't found [source]
Someone built a reproducible demonstration of this: place a secret at the very start of a 160,000-token prompt, then ask the model to repeat it back. Ollama returned HTTP 200, reported prompt_eval_count: 16387 on a 160,000-token input, and the model confidently invented an answer. No error, no warning, nothing in the logs. [source]
If you moved the directory with OLLAMA_MODELS and are not certain the server agrees with you about where it is, that is a separate failure with its own per-platform traps — see OLLAMA_MODELS not working before you go hunting for files that are not there. [source]
Evaluation basis: performance numbers below were measured on the finetuned risk model (67 held-out incidents, judge: qwen3.5, date: 2026-04-24). The GPU-served reference is serve_model.py with --quant 4bit (bitsandbytes NF4), not true fp16. Scores are cause score (max 30) and risk score (max 30); win rate is fraction of incidents ranked 1st among 5 models. [source]
q4_k_m is the recommended deployment variant. At 986 MB, it matches the larger quants on risk and comes close on summary, with the smallest footprint. [source]

Comparisons and alternatives

- `model`: name of the model to create - `from`: (optional) name of an existing model to create the new model from - `files`: (optional) a dictionary of file names to SHA256 digests of blobs to create the model from - `template`: (optional) the prompt template for the model - `renderer`: (optional) the name of the renderer for the model - `parser`: (optional) the name of the parser for the model - `license`: (optional) a string or list of strings containing the license or licenses for the model - `system`: (optional) a string containing the system prompt for the model - `parameters`: (optional [source]
Which quantization should you choose: Q4, Q5, or Q8?+ Q4_K_M by default: good quality at about 30% of FP16’s size. Move to Q5_K_M or Q6_K if memory allows, and to Q8_0 only if there is room, since the gain is very small. Going below 4 bits is justified only if the model otherwise does not fit. What does the M in Q4_K_M mean?+ M stands for Medium. The S, M, and L variants (Small, Medium, Large) differ in the share of tensors kept at higher precision. At the same bit count, M is slightly larger than S and loses slightly less quality: it is the default recommended tradeoff. How much VRAM does a Q [source]
Summary: Finetuned models are converted to GGUF and published to Ollama in three quantization variants (q4_k_m, q5_k_m, q8_0). The GPU-served reference baseline uses BnB NF4 4-bit quantization at inference (not true fp16 — the evaluation GPU lacked sufficient VRAM for the full model), so all comparisons are NF4 vs. GGUF. Quality degrades with quantization relative to the NF4 baseline: ~2% loss at q8_0, ~12% at q5_k_m, ~9% at q4_k_m. q8_0 is the best quantized variant; q5_k_m offers the best quality/size trade-off for CPU/RPi deployment. [source]
A model is a set of billions of numbers, called weights. In FP16, each occupies 16 bits: an 8-billion-parameter model weighs about 16 GB. Quantization stores each weight in fewer bits. According to the llama-quantize tool documentation, this process reduces model size and can speed up inference, at the cost of a loss of precision measured by perplexity or Kullback-Leibler divergence. The GGUF format used by llama.cpp, Ollama and LM Studio groups these variants. The right choice depends on three quantities: your card’s or Mac’s memory, the model size, and the context you plan to use. Q4_K_M off [source]
OpenBMB publishes its own GGUFs in MiniCPM5-2B-GGUF[6], with a 16-bit file in F16 rather than BF16. My three files are 2,016 bytes larger than theirs: 1,561,320,384 against 1,561,318,368 bytes for Q4_K_M. A constant gap points at the metadata, not the weights. The Ollama 0.34.1 Docker image ships its own llama-quantize in /usr/lib/ollama, built from the llama.cpp b10864 that Ollama bundles, but not the Python converter. [source]
gemma3 -it-qat tags are quantization-aware-trained Q4_0-format builds from Google. Google reports a 54% reduction in perplexity drop versus post-training Q4_0 quantization (llama.cpp perplexity eval) and weight VRAM of 54GB BF16 to 14.1GB int4 for 27B. [source]
- `format`: the format to return a response in. Format can be `json` or a JSON schema - `options`: additional model parameters listed in the documentation for the [Modelfile](./modelfile.mdx#valid-parameters-and-values) such as `temperature` - `system`: system message to (overrides what is defined in the `Modelfile`) - `template`: the prompt template to use (overrides what is defined in the `Modelfile`) - `stream`: if `false` the response will be returned as a single response object, rather than a stream of objects - `raw`: if `true` no formatting will be applied to the prompt. You may choose [source]
GGUF quantization in 2026: Q4_K_M vs Q5_K_M vs Q6_K in practice Read → Hardware · 12 min [source]
Using int4 means each number is represented using only 4 bits – a 4x reduction in data size compared to BF16. Quantization can often lead to performance degradation, so we’re excited to release Gemma 3 models that are robust to quantization. We released several quantized variants for each Gemma 3 model to enable inference with your favorite inference engine, such as Q4_0 (a common quantization format) for Ollama, llama.cpp, and MLX. [source]
Tag sets differ per model: gemma3 offers -it-qat, -it-q4_K_M, -it-q8_0, -it-fp16 and -it-bf16 (270m has no q4_K_M tag), unlike llama3.1's K-quant ladder. Check the model's tags page before assuming a quant exists. [source]
How do we maintain quality? We use QAT. Instead of just quantizing the model after it's fully trained, QAT incorporates the quantization process during training. QAT simulates low-precision operations during training to allow quantization with less degradation afterwards for smaller, faster models while maintaining accuracy. Diving deeper, we applied QAT on ~5,000 steps using probabilities from the non-quantized checkpoint as targets. We reduce the perplexity drop by 54% (using llama.cpp perplexity evaluation) when quantizing down to Q4_0. [source]
Then ollama create yourmodel-agent -f Modelfile. Note the underscore: the Modelfile parameter is num_ctx, the CLI flag is --num-ctx. Swap them and it fails silently rather than erroring. [source]
Choose Q4_K_M by default: at 4.89 bits per weight, it reduces the model size by more than threefold compared with FP16, with little quality loss. Move up to Q5_K_M or Q6_K if memory allows, use Q8_0 only if you have room left, and avoid anything below 4 bits without a good reason. What matters is your available memory: the file, context, and some headroom must fit. [source]
Evaluation basis: performance numbers below were measured on the unified model v2 (47 summary incidents, judge: gpt-oss-120b; 67 risk incidents, judge: qwen3.5; date: 2026-06-09). The fp16 baseline uses BnB NF4 4-bit quantization at inference (the evaluation GPU lacked sufficient VRAM for the full model in true fp16), so the comparison is NF4 vs. GGUF rather than full-precision vs. GGUF. [source]
Quantization does not hurt — it slightly helps on risk. Unlike the standalone risk model, all three unified GGUF variants match or exceed the fp16 baseline on risk win rate (26.9% vs 23.9%). This is explained by the NF4 vs GGUF comparison: the fp16 baseline is already a 4-bit quantized inference path, so GGUF quantization is competitive with it. [source]
q8_0 is the best quantized variant. Smallest gap vs fp16: cause 17.66 (vs 20.21), win rate 34.3% (vs 35.8%). It even outperforms fp16 on risk score and simple incident win rate (45.5% vs 38.6%). [source]
- `format`: the format to return a response in. Format can be `json` or a JSON schema. - `options`: additional model parameters listed in the documentation for the [Modelfile](./modelfile.mdx#valid-parameters-and-values) such as `temperature` - `stream`: if `false` the response will be returned as a single response object, rather than a stream of objects - `keep_alive`: controls how long the model will stay loaded into memory following the request (default: `5m`) [source]
Note the shape: expected "sha256:…", got "sha256:…" with quotes, and an HTTP 400 rather than the pull-path wording. If your error says expected/got you are looking at POST /api/blobs/:digest, which is the CLI uploading your local GGUF to the server; if it says want/got you are looking at a registry pull. [source]
Perplexity metrics rank formats consistently: more bits, less loss. But perplexity is a poor predictor of specific capabilities. A study published on arXiv in January 2026 compared llama.cpp formats on Llama-3.1-8B-Instruct, with reasoning, knowledge, and instruction-following tests: perplexity on WikiText-2 ranged from 7.32 for FP16 to 8.96 for Q3_K_S, and the authors noted that perplexity is not a complete predictor of downstream behavior. Q5_0’s average score even slightly exceeded FP16’s (69.92% versus 69.47%), which reflects measurement noise, not an improved model. [source]
The llama.cpp README measurements, taken on Llama 3.1 8B, illustrate a point the guides omit: text generation reads all active weights at every token, so it depends mainly on the number of bytes to read. In this table, generation throughput rises from about 51 tokens per second in Q8_0 to about 72 in Q4_K_M, and 90 in Q2_K_S. Prompt processing, however, remains stable at around 800 tokens per second because it is compute-bound rather than memory-bound. These figures come from specific hardware in the repository and are not representative of your machine; remember the trend: fewer bits, faster [source]
It's the maximum number of tokens the model can hold in one request: your system prompt, tool schemas, conversation history and current message combined. Current Ollama builds pick the default from VRAM (4096 under roughly 24 GiB, 32K to 48 GiB, 256K above), while the FAQ still says 4096 and the Modelfile reference still says 2048. Check yours with ollama ps rather than trusting any published figure. [source]
q5_k_m and q4_k_m both show meaningful score drops; q4_k_m surprisingly recovers slightly vs q5_k_m on simple incidents [source]
q4_k_m outperforms q5_k_m (cause 18.03 vs 16.46, win rate 26.9% vs 23.9%). q5_k_m is not a reliable quality/size middle ground for this task. [source]
q8_0 is the standout for summary. q8_0 wins 32.6% of summary incidents vs 17.0% for fp16 — a +15.6pp gain. Average score is nearly identical (5.09 vs 5.20). Ollama’s inference stack appears to produce more consistently top-ranked outputs than the BnB NF4 GPU server on this task. [source]
A digest mismatch during ollama create is a different error string from a different file, and it points at your local copy rather than the registry. The import path returns this from server/routes.go: [source]
To change where Ollama stores the downloaded models instead of using your home directory, set the environment variable `OLLAMA_MODELS` in your user account. [source]
If you want to set custom options for the model at runtime rather than in the Modelfile, you can do so with the `options` parameter. This example sets every available option, but you can set any of them individually and omit the ones you do not want to override. [source]
- `model`: name of the model to push in the form of `<namespace>/<model>:<tag>` - `insecure`: (optional) allow insecure connections to the library. Only use this if you are pushing to your library during development. - `stream`: (optional) if `false` the response will be returned as a single response object, rather than a stream of objects [source]
The llama.cpp table shows the actual savings. On Llama 3.1 8B, IQ4_XS weighs 4.17 GiB versus 4.58 GiB for Q4_K_M, or about 9% less; IQ3_XXS weighs 3.04 GiB; IQ2_XXS weighs 2.23 GiB. In the repository's measurements, generation throughput for these formats remains close to that of the K-quants, within a few tokens per second: decoding overhead is therefore modest. I-quants, however, depend on the quality of the imatrix and the model: they only make sense when Q4_K_M does not fit. [source]
1. Open Roo Code in VS Code and click the **gear icon** on the top right corner of the Roo Code window to open **Provider Settings** 2. Set `API Provider` to `Ollama` 3. (Optional) Update `Base URL` if your Ollama instance is running remotely. The default is `http://localhost:11434` 4. Enter a valid `Model ID` (for example `qwen3` or `qwen3-coder:480b-cloud`. See all supported models [here](https://ollama.com/search)) 5. Adjust the `Context Window` to at least 32K tokens for coding tasks [source]
Ollama streams several API responses as newline-delimited JSON so callers can display tokens as they arrive. Streaming lowers perceived latency for long generations, but the client must parse multiple JSON objects instead of one response body. [source]
The Ollama v0.34.1 Dockerfile[8] explains the failure: it copies the MLX build, with a CUDA 13 backend, into the amd64 image only, and the arm64 image has none. I could not test that amd64 build or the path on a Mac with Apple silicon, where MLX is native and --quantize accepts int4 or nvfp4: I have neither a Mac nor an NVIDIA GPU. If you work on a Mac and want an MLX-based alternative, read what oMLX is and how it differs from Ollama. [source]
Complex incident scores are consistent across all quants (~3.0–3.2), tied to input truncation at 4096 tokens rather than quantization effects [source]
--allow-requantize allow requantizing tensors that have already been quantized. Warning: This can severely reduce quality compared to quantizing from 16bit or 32bit [source]
--keep-split generate the quantized model in the same shards as the input file instead of a single quantized file [source]
openai is not always desired, as some tools such as Continue from VS Code do not work correctly via openai API while via ollama api they do. [source]
3. Provide your **Ollama API URL** and select your models. <Note>If you're running Onyx in Docker, to access your computer's local network use `http://host.docker.internal` instead of `http://127.0.0.1`.</Note> [source]
Non-deterministic corruption, which is almost always local rather than on the network — a network transform would repeat byte-for-byte. PR #15489 traced one such case and found the bytes arriving over HTTP were correct while re-reading the same file from disk produced a different wrong hash each time. Move the models directory to a different local disk with OLLAMA_MODELS and retry there before investigating anything else. [source]
It's important to note that while this chart uses BF16 for a fair comparison, deploying the very largest models often involves using lower-precision formats like FP8 as a practical necessity to reduce immense hardware requirements (like the number of GPUs), potentially accepting a performance trade-off for feasibility. [source]

Facts and statements

Working recipe for a non-MLX host: hf download the model, convert to BF16 GGUF with llama.cpp convert_hf_to_gguf.py, quantize with llama-quantize (e.g. Q4_K_M), then ollama create from a Modelfile pointing at the GGUF. Omit TEMPLATE if the GGUF carries its own chat template. [source]
Before v0.34.1 the import guide (v0.9.0 docs) said Ollama can quantize FP16 and FP32 models via ollama create -q/--quantize with levels such as q8_0, q4_K_S and q4_K_M; that guidance no longer applies to current versions. [source]
On 0.34.1, ollama create --quantize q4_K_M from safetensors returns 'unsupported --quantize "q4_K_M"'; the flag accepts only MLX formats int4, int8, nvfp4, mxfp4, mxfp8. From a BF16 GGUF it errors: 'create-time quantization is only supported for safetensors imports; quantize GGUF models before importing'. [source]
Current Ollama docs say 'Ollama does not quantize GGUF models during import'; quantize with a GGUF tool such as llama.cpp llama-quantize first, then use a Modelfile with FROM /path/to/file.gguf and run ollama create. [source]
Ollama v0.34.1 (published 2026-09-14) release notes: MLX safetensors ollama create is no longer experimental, and 'GGUF model creation now requires using llama.cpp tooling for safetensor conversion and quantization'. [source]
ADAPTER points at a LoRA adapter (GGUF or safetensors) and requires FROM to name the base model the adapter was tuned on; a mismatched base gives unstable behavior. The official Modelfile reference page fetched here has no ADAPTER section, and the import page does not cover it. [source]
Sources: [1] Ollama v0.34.1 release notes[12], [2] Ollama PR 14969: MLX imports and the end of GGUF conversion[1], [3] Ollama PR 18448: typical_p deprecation[3], [4] Ollama documentation: importing a model[2], [5] Ollama Modelfile reference[13], [6] Ollama v0.34.1 Dockerfile[8], [7] Ollama v0.34.2-rc0[11], [8] llama.cpp convert_hf_to_gguf.py script[14], [9] llama-quantize README[5], [10] llama.cpp Docker images[15], [11] llama.cpp v0.4.1 release notes[16], [12] MiniCPM5-2B model card on Hugging Face[4], [13] OpenBMB’s MiniCPM5-2B-GGUF[6], [14] OpenBMB guide to MiniCPM5 with Ollama[7], [15] Hub [source]
For a Hugging Face GGUF lacking a chat template, a `template` file (Go template, not Jinja), optional `system` file and JSON `params` file in the repo override the defaults; private repos work after adding the Ollama public key (~/.ollama/id_ed25519.pub) to the Hugging Face account SSH keys. [source]
FROM accepts an existing library model (name:tag), a safetensors model directory, or a GGUF file path (absolute or relative to the Modelfile). Split GGUF is supported with a wildcard (FROM ./model-*.gguf) or one FROM line per shard. [source]
For a GGUF adapter, write FROM <model name> and ADAPTER /path/to/file.gguf. FROM may be an Ollama model, a GGUF file or a Safetensors model. A GGUF adapter comes from llama.cpp's convert_lora_to_gguf.py or a download such as Hugging Face. [source]
* converting a Safetensors model with the `convert_hf_to_gguf.py` from Llama.cpp; * converting a Safetensors adapter with the `convert_lora_to_gguf.py` from Llama.cpp; or * downloading a model or adapter from a place such as HuggingFace [source]
REQUIRES <version> sets the minimum Ollama version (example 0.14.0), and a CAPABILITY instruction adds capabilities to GGUF/MLX models; the reference page also lists a draft_num_predict parameter (4 for separate draft models, 0 disables speculative drafting). These are recent additions. [source]
Split GGUF models import by keeping the original shard filenames and using a wildcard in the Modelfile, e.g. FROM /path/to/model-*.gguf. [source]
PR 14969 by Daniel Hiltgen (merged 2026-09-15) limits GGUF create to wrapping existing GGUF inputs into Ollama manifests, removes the in-tree safetensors-to-GGUF converter, the server quantization path and converter-only dependencies, adds server-side MLX safetensors imports (including remote upload/staging) and drops the --experimental CLI gate. [source]
MLX safetensors ollama create no longer experimental. GGUF model creation now requires using llama.cpp tooling for safetensor conversion and quantization. [source]
The path that does not depend on MLX is llama.cpp v0.4.1’s convert_hf_to_gguf.py and llama-quantize, followed by ollama create from the GGUF. [source]
For Llama-3.1-8B, llama.cpp lists Q4_K_M at 4.89 bits/weight (4.58GiB, 71.9 t/s generation), Q5_K_M 5.70 bpw (5.33GiB, 67.2 t/s), Q8_0 8.50 bpw (7.95GiB, 50.9 t/s), F16 16.0 bpw (14.96GiB, 29.2 t/s). Smaller quants generate faster; the README gives no perplexity figures for them. [source]
Only for MLX safetensors imports, and only with int4, int8, nvfp4, mxfp4 or mxfp8. For a GGUF, 0.34.1 rejects the flag and asks you to quantize first with llama-quantize. [source]
The Modelfile reference documents num_ctx default 2048, while the Ollama FAQ says the default context window is 4096 tokens. OLLAMA_CONTEXT_LENGTH overrides the server default; per-request num_ctx goes in the API options object, and PARAMETER num_ctx bakes it into a model. [source]
convert_hf_to_gguf.py reads the Hugging Face folder and writes a GGUF with the weights, the tokenizer and the metadata. Convert to BF16, MiniCPM5-2B’s original type, and quantize afterwards. With the full Docker image, --convert runs the script: [source]
The two model formats now have separate owners inside Ollama — MLX handles safetensors on Apple Silicon, llama.cpp handles GGUF conversion and quantization. Version 0.34.1 also improves MLX memory handling on Apple Silicon, makes model capability reporting consistent, and raises the repeat-token detection threshold to 100 tokens so it stops triggering on short repeats. [source]
Ollama exposes two API surfaces. The native one (/api/generate, /api/chat) accepts an options object containing num_ctx. The OpenAI-compatible one (/v1/chat/completions) does not. Ollama maps a fixed allowlist of OpenAI sampling parameters to model options, and num_ctx isn't on it, so the endpoint silently ignores it whether you send it nested in options or top-level. This is tracked in Ollama issue #11409 and still reproduces on current builds. [source]
SYSTEM and LICENSE take triple-quoted text (SYSTEM """..."""). LICENSE records license text in the model; the build/run commands are `ollama create <name> -f ./Modelfile` then `ollama run <name>`. [source]
Clients wanting OpenAI model names can alias a local model with `ollama cp llama3.2 gpt-3.5-turbo`. The OpenAI API has no context-size field; set num_ctx via a Modelfile PARAMETER and `ollama create`. [source]
Per the 0.9.0 import guide, Safetensors adapters are supported for Llama (2, 3, 3.1, 3.2), Mistral (1, 2, Mixtral) and Gemma (1, 2) architectures. Adapters can be produced by Hugging Face training, Unsloth or MLX. [source]
/api/generate and /api/chat stream by default; the stream is NDJSON served as application/x-ndjson, one JSON object per line. Setting "stream": false returns a single application/json object with the full text and done:true. [source]
Final-chunk usage fields: total_duration, load_duration, prompt_eval_count, prompt_eval_cached_count, prompt_eval_duration, eval_count, eval_duration. All durations are nanoseconds; counts are tokens. Usage appears only in the done:true chunk. [source]
llama-quantize shrinks the 16-bit GGUF to Q4_K_M in a single command. Model quantization with llama.cpp covers the theory. Here you pick the type in the last argument: [source]
<Tabs> <Tab title="cURL"> ```shell theme={"system"} curl http://localhost:11434/api/chat -d '{ "model": "qwen3", "messages": [{ "role": "user", "content": "How many letter r are in strawberry?" }], "think": true, "stream": false }' ``` </Tab> [source]
Make sure that you use the same base model in the `FROM` command as you used to create the adapter otherwise you will get erratic results. Most frameworks use different quantization methods, so it's best to use non-quantized (i.e. non-QLoRA) adapters. If your adapter is in the same directory as your `Modelfile`, use `ADAPTER .` to specify the adapter path. [source]
llama.cpp ships an export-lora tool that applies one or more LoRA adapters (--lora, comma-separated, or --lora-scaled FNAME:SCALE) to a base model and writes a merged GGUF, f16 by default. This is an alternative to ADAPTER: merge offline, then import the merged file with plain FROM. [source]
Ollama 0.34.1 handed GGUF conversion and quantization over to llama.cpp. The release notes sum it up in one sentence: "GGUF model creation now requires using llama.cpp tooling for safetensor conversion and quantization". The change comes from Ollama PR 14969[1], by Daniel Hiltgen, opened in March and merged hours before the release, which removes the built-in converter and server-side quantization. GitHub dates the release 14 September at 22:14 UTC, although the tag points at a commit from the 15th and the Docker image was built that same day. [source]
llama.cpp advises that requantizing an already-quantized model can severely reduce quality, so start from a 16/32-bit GGUF, and that an importance matrix can reduce accuracy loss. [source]
create-time quantization is only supported for safetensors imports; quantize GGUF models before importing: drop --quantize and run llama-quantize [source]
Once you have created your `Modelfile`, use the `ollama create` command to build the model. [source]
Now run `ollama create` from the directory where the `Modelfile` was created: [source]
```shell theme={"system"} curl -s http://localhost:11434/api/chat -H "Content-Type: application/json" -d '{ "model": "qwen3", "messages": [{"role": "user", "content": "What are the current weather conditions and temperature in New York and London?"}], "stream": false, "tools": [ { "type": "function", "function": { "name": "get_temperature", "description": "Get the current temperature for a city", "parameters": { "type": "object", "required": ["city"], "properties": { "city": {"type": "string", "description": "The name of the city"} } } } }, { "type": "function", "function": { "name": "get_cond [source]
git clone --depth 1 --branch v0.4.1 \ https://github.com/ggml-org/llama.cpp python3 -m venv .venv && . .venv/bin/activate pip install -r llama.cpp/requirements/requirements-convert_hf_to_gguf.txt python llama.cpp/convert_hf_to_gguf.py ./MiniCPM5-2B \ --outtype bf16 --outfile MiniCPM5-2B-BF16.gguf [source]
No. The ADAPTER instruction returns LoRA adapters are no longer supported and no longer appears in the Modelfile reference. The way out is to merge the adapter into the base model and convert the result to GGUF. [source]
AI/TLDR › Tools › Ollama 0.34.1 — MLX safetensors leave experimental, GGUF needs llama.cpp Ollama · 2026-09-14 · major [source]
Ollama 0.34.1 — MLX safetensors leave experimental, GGUF needs llama.cpp [source]
Ollama 0.34.1 promotes MLX safetensors model creation out of experimental and hands GGUF conversion to llama.cpp tooling. [source]
Unqualified model names resolve against defaults host registry.ollama.ai, namespace library, tag latest (defaultHost/defaultNamespace/defaultTag in the name parser). [source]
MESSAGE <role> <text> seeds conversation history; valid roles are system, user and assistant, and repeating it builds a few-shot example conversation. MESSAGE system is an alternative to SYSTEM. [source]
<Tabs> <Tab title="Ollama API"> ```shell theme={"system"} curl http://localhost:11434/api/chat \ -H "Content-Type: application/json" \ -d '{ "model": "gemma4:e2b", "messages": [ { "role": "user", "content": "Say hello in one sentence." } ], "stream": false }' ``` </Tab> [source]
Anthropic Messages compatibility is served at /v1/messages (local http://localhost:11434, cloud https://ollama.com). Clients set ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN=ollama (ignored locally). It requires Ollama v0.14.0+ (blog dated 2026-01-16). [source]
Generate chunks put text in the `response` field; chat chunks put it in `message.content`. The last chunk sets done:true and carries done_reason (e.g. "stop"). [source]
* [x] `model` * [x] `max_tokens` * [x] `messages` * [x] Text `content` * [x] Image `content` (base64) * [x] Array of content blocks * [x] `tool_use` blocks * [x] `tool_result` blocks * [x] `thinking` blocks * [x] `system` (string or array) * [x] `stream` * [x] `temperature` * [x] `top_p` * [x] `top_k` * [x] `stop_sequences` * [x] `tools` * [x] `thinking` * [x] `output_config` * [x] `effort` (model-defined names) * [ ] `tool_choice` * [ ] `metadata` [source]
After you have created a high-quality GGUF version of the model, you use llama-quantize to apply quantization. For example, quantize to Q4_K_M using a command like the following: [source]
No. Ollama 0.34.1 changes how a GGUF model is created, not how an existing one runs — safetensor conversion and quantization now go through llama.cpp tooling. Existing GGUF models also keep support for the `typical_p` parameter, which is only removed for new models, so an installed library keeps working after the upgrade. [source]
* [x] Messages * [x] Streaming * [x] System prompts * [x] Multi-turn conversations * [x] Vision (images) * [x] Tools (function calling) * [x] Tool results * [x] Thinking/extended thinking [source]
Ollama does not quantize GGUF models during import. Prepare and quantize them first with a GGUF tool such as llama.cpp's [`llama-quantize`](https://github.com/ggml-org/llama.cpp/tree/master/tools/quantize). [source]
With llama.cpp installed on the system, the equivalent is llama-quantize MiniCPM5-2B-BF16.gguf MiniCPM5-2B-Q4_K_M.gguf Q4_K_M. The end of the output sums up the reduction: [source]
unsupported --quantize "q4_K_M": supported types are int4, int8, nvfp4, mxfp4, mxfp8: the old command from safetensors; convert and quantize with llama.cpp [source]
If you are creating a model from a safetensors directory or from GGUF files, you must [push a blob](#push-a-blob) for each file and then use its file name and SHA256 digest in the `files` field. [source]
Output tokens per second = eval_count / eval_duration * 10^9 (eval_duration is nanoseconds). The docs example 18 tokens in 52,479,709 ns is about 343 tokens/s. [source]
Read the answer from `message.content`. For local inference, use `http://localhost:11434/api/chat` with a downloaded model and omit the authorization header. [source]
{ "model": "qwen3:0.6b", "created_at": "2025-05-26T03:04:17.792722554Z", "message": { "role": "assistant", "content": "<think>\n\n</think>\n\n1加1等于2。" }, "done_reason": "stop", "done": true, "total_duration": 1164766885, "load_duration": 16615755, "prompt_eval_count": 31, "prompt_eval_duration": 583130577, "eval_count": 11, "eval_duration": 562205320 } [source]
An Ollama-compatible Modelfile is written alongside the GGUF, embedding the correct template with {{ .System }}, {{ .Prompt }}, and {{ .Response }} variables, plus appropriate stop tokens (<|im_end|> and <|endoftext|> for ChatML). If an OLLAMA_README.md exists next to the script, it is copied into the output directory as README.md to populate the model card on Ollama.com. [source]
Automates the complete pipeline from raw 16-bit weights to a publicly accessible model on Ollama. For each quantization variant (default: q4_k_m, q5_k_m, q8_0), the following steps run sequentially: [source]
python convert_hf_to_gguf.py --outfile gemma-4-E2B-it-bf16.gguf --outtype bf16 --remote google/gemma-4-E2B-it [source]
python convert_hf_to_gguf.py --mmproj --outfile mmproj-gemma-4-E2B-it-Q8_0.gguf --outtype q8_0 --remote google/gemma-4-E2B-it [source]
```shell theme={"system"} curl -s http://localhost:11434/api/chat -H "Content-Type: application/json" -d '{ "model": "qwen3", "messages": [ {"role": "user", "content": "What is the temperature in New York?"}, { "role": "assistant", "tool_calls": [ { "type": "function", "function": { "index": 0, "name": "get_temperature", "arguments": {"city": "New York"} } } ] }, {"role": "tool", "tool_name": "get_temperature", "content": "22°C"} ], "stream": false }' ``` </Tab> [source]
```shell theme={"system"} curl -s http://localhost:11434/api/chat -H "Content-Type: application/json" -d '{ "model": "qwen3", "messages": [ {"role": "user", "content": "What are the current weather conditions and temperature in New York and London?"}, { "role": "assistant", "tool_calls": [ { "type": "function", "function": { "index": 0, "name": "get_temperature", "arguments": {"city": "New York"} } }, { "type": "function", "function": { "index": 1, "name": "get_conditions", "arguments": {"city": "New York"} } }, { "type": "function", "function": { "index": 2, "name": "get_temperature", "argume [source]
OpenAI-compatible base URL is http://localhost:11434/v1/ locally and https://ollama.com/v1 for cloud (OLLAMA_API_KEY). Locally the client must send some API key value, but Ollama ignores it. [source]
/v1/responses (added in v0.13.3) supports streaming, function tools and reasoning summaries but only the non-stateful flavor: previous_response_id, conversation and truncation are unavailable. Cloud also lacks built-in web search through it. [source]
Ollama uses the Jinja chat template that llama.cpp stores in the GGUF: the imported model advertised tools and thinking without a TEMPLATE line. [source]
We want you to be able to use these models easily within your preferred workflow. Our official int4 and Q4_0 unquantized QAT models are available on Hugging Face and Kaggle. We’ve partnered with popular developer tools that enable seamlessly trying out the QAT-based quantized checkpoints: [source]
<Tabs> <Tab title="cURL"> ```shell theme={"system"} curl -X POST http://localhost:11434/api/chat -H "Content-Type: application/json" -d '{ "model": "gpt-oss", "messages": [{"role": "user", "content": "Tell me about Canada in one line"}], "stream": false, "format": "json" }' ``` </Tab> [source]
<Tabs> <Tab title="cURL"> ```shell theme={"system"} curl -X POST http://localhost:11434/api/chat -H "Content-Type: application/json" -d '{ "model": "gpt-oss", "messages": [{"role": "user", "content": "Tell me about Canada."}], "stream": false, "format": { "type": "object", "properties": { "name": {"type": "string"}, "capital": {"type": "string"}, "languages": { "type": "array", "items": {"type": "string"} } }, "required": ["name", "capital", "languages"] } }' ``` </Tab> [source]
```shell basic.sh theme={"system"} curl -X POST http://localhost:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-oss:20b", "messages": [{ "role": "user", "content": "Say this is a test" }] }' ``` </CodeGroup> [source]
The FROM base must be the same base model used to train the adapter. Otherwise results are erratic. The same rule is stated for GGUF adapters. [source]
* a model from Ollama * a GGUF file * a Safetensors based model [source]
Runs ollama create <model-name>:<quant> -f Modelfile from within the output directory. This registers the model in the local Ollama registry, making it immediately usable via ollama run without a network round-trip. [source]
<Tabs> <Tab title="cURL"> ```shell theme={"system"} curl -s http://localhost:11434/api/chat -H "Content-Type: application/json" -d '{ "model": "qwen3", "messages": [{"role": "user", "content": "What is the temperature in New York?"}], "stream": false, "tools": [ { "type": "function", "function": { "name": "get_temperature", "description": "Get the current temperature for a city", "parameters": { "type": "object", "required": ["city"], "properties": { "city": {"type": "string", "description": "The name of the city"} } } } } ] }' ``` [source]
Copying only manifests or only blobs between machines breaks the model; both subdirectories must travel together. [source]
If you're using the API, use the `keep_alive` parameter with the `/api/generate` and `/api/chat` endpoints to set the amount of time that a model stays in memory. The `keep_alive` parameter can be set to: [source]
The `keep_alive` API parameter with the `/api/generate` and `/api/chat` API endpoints will override the `OLLAMA_KEEP_ALIVE` setting. [source]
Handle chat streaming from message.content chunks. $ curl http://localhost:11434/api/chat -d '{"model":"gpt-oss:20b","messages":[{"role":"user","content":"Return OK."}]}' {"message":{"content":"O"},"done":false} {"message":{"content":"K"},"done":false} {"done":true} [source]
First, create a Modelfile with the FP16 or FP32 based model you wish to quantize. [source]
On Hugging Face and in Ollama, the same model exists in dozens of variants: Q4_K_M, Q5_K_S, Q6_K, Q8_0, IQ3_XXS, FP16. This guide gives you measured sizes, a calculation for estimating the memory of any model, and a decision rule based on your VRAM. [source]
How to quantize the GGUF with llama-quantize [source]
<Tabs> <Tab title="Ollama API"> ```shell theme={"system"} curl https://ollama.com/api/chat \ -H "Authorization: Bearer $OLLAMA_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gemma4:31b", "messages": [ { "role": "user", "content": "Say hello in one sentence." } ], "stream": false }' ``` [source]
必須命令はFROMのみ。他の7命令（PARAMETER / TEMPLATE / SYSTEM / ADAPTER / LICENSE / MESSAGE / REQUIRES）は任意 [source]
First, create a `Modelfile` with a `FROM` command pointing at the base model you used for fine tuning, and an `ADAPTER` command which points to the directory with your Safetensors adapter: [source]
For a GGUF adapter, create the `Modelfile` with: [source]
#gguf #hugging-face #llama-cpp #local-ai #ollama #quantization [source]
In Ollama 0.34.1, ollama create --quantize q4_K_M answers unsupported --quantize "q4_K_M": the flag only accepts the MLX formats int4, int8, nvfp4, mxfp4 and mxfp8. [source]
ADAPTER in the Modelfile Reads the adapter file LoRA adapters are no longer supported [source]
curl http://localhost:11434/api/chat -d '{ "model": "deepseek-r1", "messages": [ { "role": "user", "content": "how many r in the word strawberry?" } ], "think": true, "stream": false }' [source]
If your framework points at http://localhost:11434/v1, passing num_ctx in the request body does nothing. Set OLLAMA_CONTEXT_LENGTH on the server, or bake num_ctx into a Modelfile, so the value applies before the request arrives. [source]
Beyond making MLX safetensors creation stable, Ollama 0.34.1 improves MLX memory handling on Apple Silicon and updates both the MLX and llama.cpp dependencies. Model capability reporting is also made consistent, so a client asking what a model supports gets the same answer regardless of which of the two backends is serving it. [source]
Creating or deleting models requires a local Ollama server. [System One](/capabilities/decision) also requires a local server and is not yet available in Ollama's cloud. For supported API features, see [OpenAI compatibility](/api/openai-compatibility) and [Anthropic compatibility](/api/anthropic-compatibility). [source]
Ollama can quantize FP16 and FP32 based models into different quantization levels using the `-q/--quantize` flag with the `ollama create` command. [source]
How to convert the safetensors to GGUF with llama.cpp [source]
The script stores the chat template from chat_template.jinja and the sampling defaults from generation_config.json in the GGUF: temperature 1.0 and top_p 0.95. If a model needs transformers 5, the llama-quantize documentation[5] says you can upgrade it with pip install -U transformers; MiniCPM5-2B did not need it. [source]
The `files` parameter should include the file name and SHA256 digest of each file in the Safetensors model. Use [/api/blobs/:digest](#push-a-blob) to push each file before calling this API. Files remain in the cache until the Ollama server is restarted. [source]
ollama create mymodel -f Modelfile -q q4_K_M [source]
ollama create small-model -f Modelfile -q q4_K_M [source]
```shell basic.sh theme={"system"} curl -X POST http://localhost:11434/v1/messages \ -H "Content-Type: application/json" \ -H "x-api-key: ollama" \ -H "anthropic-version: 2023-06-01" \ -d '{ "model": "qwen3-coder", "max_tokens": 1024, "messages": [{ "role": "user", "content": "Hello, how are you?" }] }' ``` </CodeGroup> [source]
* `f16` - high precision and memory usage (default). * `q8_0` - 8-bit quantization, uses approximately 1/2 the memory of `f16` with a very small loss in precision, this usually has no noticeable impact on the model's quality (recommended if not using f16). * `q4_0` - 4-bit quantization, uses approximately 1/4 the memory of `f16` with a small-medium loss in precision that may be more noticeable at higher context sizes. [source]
Send a default streaming generate request. $ curl http://localhost:11434/api/generate -d '{ "model": "gpt-oss:20b", "prompt": "Return the word OK." }' {"model":"gpt-oss:20b","response":"O","done":false} {"model":"gpt-oss:20b","response":"K","done":false} {"model":"gpt-oss:20b","done":true} [source]
Contents Quantization: what you trade for memoryDecoding the names: Q4_K_M, Q5_K_S, Q8_0, IQ3_XXSDecision table: measured sizes and lossesCalculate a model's memory in secondsWhat quantization really does to qualityWhy a smaller file also generates fasterChoose using three rulesI-quants and sub-4-bit formatsModels published directly in 4-bitWhat to choose based on your memory [source]
FROM ./MiniCPM5-2B-Q4_K_M.gguf PARAMETER num_ctx 8192 PARAMETER num_thread 4 SYSTEM """Responde en español de España.""" [source]
Model architecture llama parameters 2.5B context length 131072 embedding length 2048 quantization Q4_K_M [source]
Yes. ollama create copies the file into its blob store, which lives in /root/.ollama/models/blobs in the Docker image. The model keeps working without the original GGUF, and ollama rm frees that copy. [source]
Show information about a model including details, modelfile, template, parameters, license, system prompt. [source]
./build/bin/llama-quantize gemma-4-E2B-it-bf16.gguf gemma-4-E2B-it-Q4_K_M.gguf Q4_K_M [source]
./llama-quantize input-model-f32.gguf q4_k_m 8 [source]
./llama-quantize --allow-requantize --leave-output-tensor --pure input-model-f32.gguf q4_k_m 8 [source]
./llama-quantize --imatrix imatrix.gguf --include-weights attn_v --include-weights ffn_down input-model-f32.gguf q4_k_m 8 [source]
./llama-quantize --imatrix imatrix.gguf --output-tensor-type q5_k --token-embedding-type q3_k --keep-split input-model-f32.gguf q4_k_m 8 [source]
./llama-quantize --imatrix imatrix.gguf --tensor-type "\.(\d*[13579])\.attn_k=q5_k" --tensor-type "\.(\d*[02468])\.attn_q=q3_k" input-model-f32.gguf q4_k_m 8 [source]
curl -X POST http://localhost:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "gpt-oss:20b", "messages": [{ "role": "user", "content": "Say this is a test" }] }' [source]
By default, a template will be selected automatically from a list of commonly used templates. It will be selected based on the built-in `tokenizer.chat_template` metadata stored inside the GGUF file. [source]
Our official Quantization Aware Trained (QAT) models provide a high-quality baseline, but the vibrant Gemmaverse offers many alternatives. These often use Post-Training Quantization (PTQ), with significant contributions from members such as Bartowski, Unsloth, and GGML readily available on Hugging Face. Exploring these community options provides a wider spectrum of size, speed, and quality trade-offs to fit specific needs. [source]
Both of Ollama’s generate API (/api/generate) and chat API (/api/chat) have been updated to support thinking. [source]
If you create the Modelfile in the same directory as the weights, you can use the command `FROM .`. [source]
1. From `Files and versions` tab on a model page, open GGUF viewer on a particular GGUF file. 2. Choose `ollama` from `Use this model` dropdown. [source]
curl http://localhost:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "llama2", "messages": [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": "Hello!" } ] }' [source]
Can I import a LoRA adapter in Ollama 0.34.1? [source]
docker run --rm -u "$(id -u):$(id -g)" -e HOME=/tmp \ -v "$PWD:/models" \ ghcr.io/ggml-org/llama.cpp:full-v0.4.1 \ --convert /models/MiniCPM5-2B --outtype bf16 \ --outfile /models/MiniCPM5-2B-BF16.gguf [source]
docker run --rm -u "$(id -u):$(id -g)" -v "$PWD:/models" \ ghcr.io/ggml-org/llama.cpp:full-v0.4.1 \ --quantize /models/MiniCPM5-2B-BF16.gguf \ /models/MiniCPM5-2B-Q4_K_M.gguf Q4_K_M [source]
Importing costs no speed against llama.cpp. I compared Ollama with llama.cpp’s server-v0.4.1 image on the same GGUF, with 4 threads, temperature 0, 128 tokens and thinking off, over five alternating rounds: [source]
Ollama's own documentation currently disagrees with itself. The FAQ says the default is 4096. The Modelfile reference says num_ctx defaults to 2048. The context length page describes a default derived from available VRAM: 4K below 24 GiB, 32K from 24 to 48 GiB, and 256K above that. Each of those was true of some build, and the VRAM-derived version is what current releases actually do. [source]
# 3. Send it to Ollama curl -X POST http://localhost:11434/api/chat \ -H "Content-Type: application/json" \ -d '{ "model": "gemma4", "messages": [{ "role": "user", "content": "What is in this image?", "images": ["'"$IMG"'"] }], "stream": false }' ``` </Tab> [source]
<Tab title="OpenAI Chat Completions"> ```shell theme={"system"} curl http://localhost:11434/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "gemma4:e2b", "messages": [ { "role": "user", "content": "Say hello in one sentence." } ] }' ``` [source]
client = anthropic.Anthropic( base_url='http://localhost:11434', api_key='ollama', # required but ignored ) [source]
const anthropic = new Anthropic({ baseURL: "http://localhost:11434", apiKey: "ollama", // required but ignored }); [source]
```shell responses.sh theme={"system"} curl -X POST http://localhost:11434/v1/responses \ -H "Content-Type: application/json" \ -d '{ "model": "qwen3:8b", "input": "Write a short poem about the color blue" }' ``` </CodeGroup> [source]
# append accumulated fields to the messages if thinking or content or tool_calls: messages.append({'role': 'assistant', 'thinking': thinking, 'content': content, 'tool_calls': tool_calls}) [source]
if (thinking || content || toolCalls.length) { messages.push({ role: 'assistant', thinking, content, tool_calls: toolCalls } as any) } [source]
On Linux without MLX, importing safetensors fails with MLX runtime is not available, and ADAPTER and typical_p are no longer accepted. [source]
- `options`: additional model parameters listed in the documentation for the [Modelfile](./modelfile.mdx#valid-parameters-and-values) such as `temperature` - `keep_alive`: controls how long the model will stay loaded into memory following the request (default: `5m`) [source]
Measure Q4_K_S Q4_K_M Q5_K_S Q5_K_M Q6_K Q8_0 [source]
Short: -qQuantization level for the model (reduces size and memory usage)Options: q4_K_M, q4_K_S, q5_K_M, q5_K_S, q8_0 [source]
* [x] `id` * [x] `type` * [x] `role` * [x] `model` * [x] `content` (text, tool\_use, thinking blocks) * [x] `stop_reason` (end\_turn, max\_tokens, tool\_use) * [x] `usage` (input\_tokens, output\_tokens) [source]
* [x] Chat completions * [x] Streaming * [x] JSON mode * [x] Reproducible outputs * [x] Vision * [x] Tools * [x] Reasoning/thinking control (for thinking models) * [ ] Logprobs [source]
Given the safetensors CreateModel works both local+remote, does it make sense to do the same for stitching together the GGUFs? That could potentially save transferring a lot of data. [source]
@@ -1196,15 +1197,17 @@ If you are creating a model from a safetensors directory or from a GGUF file, yo [source]
| Instruction | Description | | - | - | | [`FROM`](#from-required) (required) | Defines the base model to use. | | [`PARAMETER`](#parameter) | Sets the parameters for how Ollama will run the model. | | [`TEMPLATE`](#template) | The full prompt template to be sent to the model. | | [`SYSTEM`](#system) | Specifies the system message that will be set in the template. | | [`LICENSE`](#license) | Specifies the legal license. | | [`MESSAGE`](#message) | Specify message history. | | [`REQUIRES`](#requires) | Specify the minimum version of Ollama required by the model. | | [`CAPABILITY`](#capability) [source]
The GGUF file location should be specified as an absolute path or relative to the `Modelfile` location. [source]
GGUF, safetensors: understanding the formats [source]
Cleanup — the intermediate F16 GGUF and .imatrix.dat files are deleted; only the final quantized GGUF is kept [source]
Calls convert_to_gguf.py with the model path and target quant, writing the GGUF file, Modelfile, and README to ./gguf_<quant>/. [source]
- another model; - a safetensors directory; or - one or more GGUF files. [source]
The GGUF file location should be specified as an absolute path or relative to the Modelfile location. For a split GGUF model, keep the original split filenames and use a wildcard that matches every shard. Shards can also be listed with separate FROM instructions. [source]
Quantize the converted GGUF file. [source]
OLLAMA_REGISTRY_MAXSTREAMS=1 no longer exists. It appeared in the same suggestion, and a search of the current repository returns no occurrences of that name in any file. The nearest surviving relative is OLLAMA_MAX_TRANSFER_STREAMS (default 4), and its own comment in envconfig/config.go scopes it explicitly: it "caps the number of simultaneous body-bearing transfers during safetensors model pulls/pushes" and "has no effect on GGUF transfers, which use the legacy upload/download paths." Setting it will not reduce the parallel range requests that a normal GGUF pull uses. [source]
You can also use [OpenAI](/api/openai-compatibility) or [Anthropic](/api/anthropic-compatibility) clients. Each supports a subset of the original API. No Ollama installation is required. [source]
response = chat( model='qwen3', messages=[{'role': 'user', 'content': 'How many letter r are in strawberry?'}], think=True, stream=False, ) [source]
const response = await ollama.chat({ model: 'deepseek-r1', messages: [{ role: 'user', content: 'How many letter r are in strawberry?' }], think: true, stream: false, }) [source]
- `model`: (required) the [model name](#model-names) - `messages`: the messages of the chat, this can be used to keep a chat memory - `tools`: list of tools in JSON for the model to use if supported - `think`: (for thinking models) should the model think before responding? Can be a boolean or a thinking level (`"low"`, `"medium"`, `"high"`, or `"max"`). [source]
| Feature | Status | | - | - | | Image content | Base64 images supported; URL images not supported | | Extended thinking | Basic support; `budget_tokens` accepted but not enforced | [source]
async function agentLoop() { const messages = [{ role: 'user', content: 'What is (11434+12341)*412?' }] [source]
llama.cpp convert_hf_to_gguf.py script [source]
Certain API endpoints stream responses by default, such as `/api/generate`. These responses are provided in the newline-delimited JSON format (i.e. the `application/x-ndjson` content type). For example: [source]
1. Chatting: Stream partial assistant messages. Each chunk includes the `content` so you can render messages as they arrive. 2. Thinking: Thinking-capable models emit a `thinking` field alongside regular content in each chunk. Detect this field in streaming chunks to show or hide reasoning traces before the final answer arrives. 3. Tool calling: Watch for streamed `tool_calls` in each chunk, execute the requested tool, and append tool outputs back into the conversation. [source]
done_thinking = False # accumulate the partial fields for chunk in stream: if chunk.message.thinking: thinking += chunk.message.thinking print(chunk.message.thinking, end='', flush=True) if chunk.message.content: if not done_thinking: done_thinking = True print('\n') content += chunk.message.content print(chunk.message.content, end='', flush=True) if chunk.message.tool_calls: tool_calls.extend(chunk.message.tool_calls) print(chunk.message.tool_calls) [source]
client = OpenAI( base_url='http://localhost:11434/v1/', api_key='ollama', # required but ignored ) [source]
const openai = new OpenAI({ baseURL: "http://localhost:11434/v1/", apiKey: "ollama", // required but ignored }); [source]
chat_completion = client.chat.completions.create( messages=[ { 'role': 'user', 'content': 'Say this is a test', } ], model='gpt-oss:20b', ) print(chat_completion.choices[0].message.content) ``` [source]
const chatCompletion = await openai.chat.completions.create({ messages: [{ role: "user", content: "Say this is a test" }], model: "gpt-oss:20b", }); [source]
messages.push(response.message) console.log('Thinking:', response.message.thinking) console.log('Content:', response.message.content) [source]
while True: stream = chat( model='qwen3', messages=messages, tools=[get_temperature], stream=True, think=True, ) [source]
thinking = '' content = '' tool_calls = [] [source]
while (true) { const stream = await ollama.chat({ model: 'qwen3', messages, tools: [getTemperatureTool], stream: true, think: true, }) [source]
let thinking = '' let content = '' const toolCalls: any[] = [] let doneThinking = false [source]
ollama create choose-a-model-name -f ./Modelfile ollama run choose-a-model-name [source]
If you create the Modelfile in the same directory as the weights, you can use the command FROM .. Now run the ollama create command from the directory where you created the Modelfile: [source]
1. Save it as a file (e.g. `Modelfile`) 2. `ollama create choose-a-model-name -f <location of the file e.g. ./Modelfile>` 3. `ollama run choose-a-model-name` 4. Start using the model! [source]
Adds a capability to a GGUF or MLX model without removing inherited or inferred capabilities. Repeat the instruction to add more than one. Declare `decision` only for weights trained for System One's decision prompts: [source]
The gpt-oss models are reasoning models. By default, the model will do medium level reasoning. To control the reasoning you can specify in the system message the reasoning level as low, medium, or high. The recommended format is: [source]
Why you do not need TEMPLATE in the Modelfile [source]
Can I still quantize with ollama create --quantize? [source]
ollama create minicpm5-2b ollama show minicpm5-2b ollama run minicpm5-2b --think=false --verbose \ "Resume en una frase qué es un Modelfile de Ollama." [source]
Can I still quantize with ollama create –quantize? [source]
So skip the lookup tables. The default is a property of your Ollama version and machine, not of the model you pulled, unless that model's own Modelfile sets num_ctx. Run ollama show yourmodel, then ollama ps. Two commands beat any list. [source]
ollama create MODEL ollama create MODEL -f MODELFILE [source]
ollama create mymodel -f ./custom/Modelfile [source]
ollama create code-assistant -f Modelfile [source]
ollama create medical-assistant -f Modelfile [source]
ollama create mymodel -f /path/to/Modelfile [source]
ollama pull llama3.2 ollama create mymodel -f Modelfile [source]
What about ollama create and GGUF imports? [source]
| API | Direct cloud access | Local Ollama server | | - | - | - | | Ollama | `https://ollama.com/api` | `http://localhost:11434/api` | | OpenAI compatibility | `https://ollama.com/v1` | `http://localhost:11434/v1` | | Anthropic client base URL | `https://ollama.com` | `http://localhost:11434` | [source]
If the model resolves named levels from `/api/show` metadata, Ollama applies supported names exactly. Unsupported names use the model default. [source]
- `digest`: the SHA256 digest of the blob [source]
- `digest`: the expected SHA256 digest of the file [source]
Home/Guides/Getting started/Configuration/Choose your quantization (Q4, Q5, Q8, FP16) [source]
Choose your quantization (Q4, Q5, Q8, FP16) [source]
Some recent models are no longer trained in 16-bit and then compressed: their weights are produced in 4-bit with quantization-aware training. Kimi K3 is an example: its specifications list MXFP4 weights and MXFP8 activations, with quantization-aware training. For these models, the native file is already the reference, and further quantization to a lower format degrades it more than quantizing a 16-bit model. Always read the model specifications before converting it. [source]
Re-quantize with importance guidance — llama-quantize --imatrix performs non-uniform quantization: weights more important to the model’s predictions are preserved at higher precision, while less critical weights are compressed more aggressively [source]
q8_0 quality loss is negligible (−0.11) — the best quantized variant by a clear margin [source]
FROM ~/.ollama/models/blobs/sha256-00e1317cbf74d901080d7100f57580ba8dd8de57203072dc6f668324ba545f29 TEMPLATE """{{ if .System }}<|start_header_id|>system<|end_header_id|> [source]
Model Original size Quantized size (Q4_K_M) [source]
Take the digest after want, replace the colon with a hyphen, and look for that filename in the blobs directory. That is not a heuristic; it is literally what manifest.BlobsPath() does: [source]
What it changes is startup housekeeping. On boot, Serve() runs PruneLayers() unless the variable is set, and that function walks blobs/, converts - back to : in each filename, and removes any file that will not parse as a valid digest — the source comment reads "remove invalid blobs (e.g. partial downloads)" — subject to a one-hour grace period on the file's modification time. Anything left over that no manifest references then gets deleted too. [source]
```shell streaming.sh theme={"system"} curl -X POST http://localhost:11434/v1/messages \ -H "Content-Type: application/json" \ -d '{ "model": "qwen3-coder", "max_tokens": 1024, "stream": true, "messages": [{ "role": "user", "content": "Count from 1 to 10" }] }' ``` </CodeGroup> [source]
client = anthropic.Anthropic( base_url='http://localhost:11434', api_key='ollama', ) [source]
- `model`: (required) the [model name](#model-names) - `prompt`: the prompt to generate a response for - `suffix`: the text after the model response - `images`: (optional) a list of base64-encoded images (for multimodal models such as `llava`) - `think`: (for thinking models) should the model think before responding? Can be a boolean or a thinking level (`"low"`, `"medium"`, `"high"`, or `"max"`). [source]
You can optionally configure a system prompt by putting it into a new file named `system` in the repository. [source]
const anthropic = new Anthropic({ baseURL: "http://localhost:11434", apiKey: "ollama", }); [source]
<CodeGroup> ```python basic.py theme={"system"} import anthropic [source]
```javascript basic.js theme={"system"} import Anthropic from "@anthropic-ai/sdk"; [source]
If you are using the API you can preload a model by sending the Ollama server an empty request. This works with both the `/api/generate` and `/api/chat` API endpoints. [source]
FROM llama3.2 ADAPTER ./ollama-lora.gguf [source]
POST http://localhost:11434/api/chat [source]
Apply LORA adapters to base model and export the resulting model. [source]
Multiple LORA adapters can be applied by passing comma-separated values to `--lora FNAME` or `--lora-scaled FNAME:SCALE,...`: [source]
* `total_duration`: How long the response took to generate * `load_duration`: How long the model took to load * `prompt_eval_count`: How many input tokens were in the prompt * `prompt_eval_cached_count`: How many prompt tokens were read from the cache * `prompt_eval_duration`: How long it took to evaluate the uncached prompt tokens * `eval_count`: How many output tokens were processes * `eval_duration`: How long it took to generate the output tokens [source]
What changed about importing models in Ollama 0.34.1 [source]
I ran the same commands on the previous release, 0.34.0, and on 0.34.1, against the MiniCPM5-2B folder and its GGUF. This is what each one answered: [source]
You can also run private GGUFs from the Hugging Face Hub. [source]
1. Copy your Ollama SSH key, you can do so via: `cat ~/.ollama/id_ed25519.pub | pbcopy` 2. Add the corresponding key to your Hugging Face account by going to [your account settings](https://huggingface.co/settings/keys) and clicking on `Add new SSH key`. 3. That's it! You can now run private GGUFs from the Hugging Face Hub: `ollama run hf.co/{username}/{repository}`. [source]
- https://github.com/ollama/ollama/blob/main/docs/README.md - https://huggingface.co/docs/hub/en/gguf [source]
Does Ollama 0.34.1 break GGUF models I already have? [source]
* "https://github.com/ollama/ollama/blob/main/docs/modelfile.md#valid-parameters-and-values">Ollama [source]
* [x] Streaming * [x] Tools (function calling) * [x] Reasoning summaries (for thinking models) * [ ] Stateful requests [source]
<Tab title="JavaScript"> ```javascript theme={"system"} import ollama from 'ollama' [source]
<Tabs> <Tab title="Python"> ```python theme={"system"} from ollama import chat [source]
if chunk.message.thinking: print(chunk.message.thinking, end='') elif chunk.message.content: if in_thinking: print('\n\nAnswer:\n', end='') in_thinking = False print(chunk.message.content, end='') [source]
* The local server does not validate API keys. Direct cloud inference requires a valid bearer token. * The hosted endpoint rejects `anthropic-version` values older than `2023-06-01`. Use `2023-06-01` in requests. * Token counts are approximations based on the underlying model's tokenizer [source]
# append the accumulated fields to the messages for the next request new_messages = [{ role: 'assistant', thinking: thinking, content: content }] ``` </Tab> [source]
// append the accumulated fields to the messages for the next request new_messages = [{ role: 'assistant', thinking: thinking, content: content }] } [source]
# The python client automatically parses functions as a tool schema so we can pass them directly # Schemas can be passed directly in the tools list as well response = chat(model='qwen3', messages=messages, tools=[get_temperature, get_conditions], think=True) [source]
# generate the final response final_response = chat(model='qwen3', messages=messages, tools=[get_temperature, get_conditions], think=True) print(final_response.message.content) ``` </Tab> [source]
const response = await ollama.chat({ model: 'qwen3', messages, tools, think: true }) [source]
// generate the final response const finalResponse = await ollama.chat({ model: 'qwen3', messages, tools, think: true }) console.log(finalResponse.message.content) } ``` </Tab> </Tabs> [source]
while (true) { const response = await ollama.chat({ model: 'qwen3', messages, tools, think: true, }) [source]
Thinking controls vary by model. Use `/api/show` to discover the values a model supports and the value Ollama uses by default: [source]
* `true`: request thinking output. * `false`: request no thinking output, if the model permits it. * `null`: use the model default. * A string: select a supported level from `thinking.values`. Use the exact value from `/api/show`. Numbers are not supported. [source]
create: add server-side MLX imports and drop GGUF conversion - #14969 #14969 [source]
create: add server-side MLX imports and drop GGUF conversion#14969 [source]
create: run MLX quantization on worker thread pd95/ollama#11 [source]
create: add server-side MLX imports and drop GGUF conversion [source]
Source: arXiv study on llama.cpp quantization (2026) [source]
Source: llama.cpp discussion on quantization methods [source]
msg="template selection" model=…/minicpm5-2b:openbmb-template selected=gguf_chat_template go_template=[completion] chat_template="[tools thinking completion]" [source]
GGUF creationNow needs llama.cpp tooling [source]
Read text blocks from content. The cloud endpoint requires Authorization: Bearer; it does not accept x-api-key alone. Compatible models support messages, streaming, and function calling. Tool-choice controls, deferred tools, and hosted web search are not fully supported. [source]
Web search is provided as a REST API with deeper tool integrations in the Python and JavaScript libraries. This also enables models like OpenAI’s gpt-oss models to conduct long-running research tasks. [source]
for call in tool_calls: if call.function.name == 'get_temperature': result = get_temperature(**call.function.arguments) else: result = 'Unknown tool' messages.append({'role': 'tool', 'tool_name': call.function.name, 'content': result}) ``` </Tab> [source]
const getTemperatureTool = { type: 'function', function: { name: 'get_temperature', description: 'Get the current temperature for a city', parameters: { type: 'object', required: ['city'], properties: { city: { type: 'string', description: 'The name of the city' }, }, }, }, } [source]
async function agentLoop() { const messages = [{ role: 'user', content: "What is the temperature in New York?" }] [source]
for (const call of toolCalls) { if (call.function.name === 'get_temperature') { const args = call.function.arguments as { city: string } const result = getTemperature(args.city) messages.push({ role: 'tool', tool_name: call.function.name, content: result } ) } else { messages.push({ role: 'tool', tool_name: call.function.name, content: 'Unknown tool' } ) } } } } [source]
TEMPLATE """{{ if .System }}<|start_header_id|>system<|end_header_id|> {{ .System }}<|eot_id|>{{ end }}{{ if .Prompt }}<|start_header_id|>user<|end_header_id|> {{ .Prompt }}<|eot_id|>{{ end }}<|start_header_id|>assistant<|end_header_id|> {{ .Response }}<|eot_id|>""" PARAMETER stop "<|start_header_id|>" PARAMETER stop "<|end_header_id|>" PARAMETER stop "<|eot_id|>" PARAMETER stop "<|reserved_special_token" [source]
`TEMPLATE` of the full prompt template to be passed into the model. It may include (optionally) a system message, a user's message and the response from the model. Note: syntax may be model specific. Templates use Go [template syntax](https://pkg.go.dev/text/template). [source]
The `SYSTEM` instruction specifies the system message to be used in the template, if applicable. [source]
#Decoding the names: Q4_K_M, Q5_K_S, Q8_0, IQ3_XXS [source]
Older formats with a uniform precision level. Q8_0 is still used for its near-zero loss, but Q4_0 and Q5_0 have been superseded by the K variants. [source]
Choose a _K_M suffix: Q4_K_M, Q5_K_M, or Q6_K. Reserve IQ and _XXS for cases where the model otherwise will not fit. [source]
q4_k_m — outperforms q5_k_m on this task [source]
The num_ctx parameter raises the context above the 4096 tokens Ollama assigns by default on a machine without a GPU, according to its own log. The num_thread parameter caps the CPU threads, and the performance section explains why I added it. The SYSTEM line sets the answer language to peninsular Spanish. Then create and check the model: [source]
If the messages array is empty and the `keep_alive` parameter is set to `0`, a model will be unloaded from memory. [source]
SYSTEMSpecifies the system message that will be set in the template. [source]
Yes. Hermes sends num_ctx but talks to local Ollama over /v1, so the value is ignored and the model loads at the server default. A native adapter is in progress upstream. Until it ships, set OLLAMA_CONTEXT_LENGTH on the server; the Hermes-side context setting alone will not fix it. [source]
curl -X POST http://localhost:11434/v1/messages \ -H "Content-Type: application/json" \ -d '{ "model": "qwen3-coder", "max_tokens": 1024, "stream": true, "messages": [{ "role": "user", "content": "Count from 1 to 10" }] }' [source]
Issue #9846 is titled with that exact digest — a Windows user on 0.6.1 whose got was e3b0c442…, with a server log showing POST "/api/pull" returning 200 in 5.76 seconds. A multi-gigabyte model does not download in under six seconds. Nothing was downloaded at all. [source]
* [x] `model` * [x] `input` * [x] `instructions` * [x] `tools` * [x] `stream` * [x] `temperature` * [x] `top_p` * [x] `max_output_tokens` * [x] `reasoning` * [x] `effort` (model-defined names and compatibility aliases) * [x] `think` (Ollama extension; boolean, model-defined string, or `null`) * [ ] `previous_response_id` (stateful v1/responses not supported) * [ ] `conversation` (stateful v1/responses not supported) * [ ] `truncation` [source]
<Tab title="OpenAI Chat Completions"> ```shell theme={"system"} curl https://ollama.com/v1/chat/completions \ -H "Authorization: Bearer $OLLAMA_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "model": "gemma4:31b", "messages": [ { "role": "user", "content": "Say hello in one sentence." } ] }' ``` [source]
<Tab title="Anthropic Messages"> ```shell theme={"system"} curl http://localhost:11434/v1/messages \ -H "Content-Type: application/json" \ -H "anthropic-version: 2023-06-01" \ -d '{ "model": "gemma4:e2b", "max_tokens": 1024, "messages": [ { "role": "user", "content": "Say hello in one sentence." } ] }' ``` [source]
CLI controls from the launch blog: `--think` enables, `--think=false` disables, `--hidethinking` hides reasoning for scripting, and `/set think` / `/set nothink` toggle inside an interactive session. [source]
Before the think switch shipped, Qwen3 on Ollama ignored `enable_thinking=False` and users had to append `/no_think` to prompts; a maintainer closed the issue on 2025-05-30 pointing to PR #10584 and the v0.9.0 release as the configurable switch. [source]
message = client.messages.create( model='qwen3-coder', max_tokens=1024, messages=[ {'role': 'user', 'content': 'Hello, how are you?'} ] ) print(message.content[0].text) ``` [source]
const message = await anthropic.messages.create({ model: "qwen3-coder", max_tokens: 1024, messages: [{ role: "user", content: "Hello, how are you?" }], }); [source]
console.log(message.content[0].text); ``` [source]
<CodeGroup> ```python basic.py theme={"system"} from openai import OpenAI [source]
```javascript basic.js theme={"system"} import OpenAI from "openai"; [source]
console.log(chatCompletion.choices[0].message.content); ``` [source]
<CodeGroup> ```python responses.py theme={"system"} from openai import OpenAI [source]
```javascript responses.js theme={"system"} import OpenAI from "openai"; [source]
<CodeGroup> ```python vision.py theme={"system"} from openai import OpenAI [source]
```javascript vision.js theme={"system"} import OpenAI from "openai"; [source]
3. Edit or create a new variable for your user account for `OLLAMA_MODELS` where you want the models stored [source]
Thinking streams interleave reasoning tokens before answer tokens. Detect the first `thinking` chunk to render a "thinking" section, then switch to the final reply once `message.content` arrives. [source]
for chunk in stream: if chunk.message.thinking and not in_thinking: in_thinking = True print('Thinking:\n', end='') [source]
- `parameters`: (optional) a dictionary of parameters for the model (see [Modelfile](./modelfile.mdx#valid-parameters-and-values) for a list of parameters) [source]
Ollama v0.14.0 and later are now compatible with the Anthropic Messages API, making it possible to use tools like Claude Code with open-source models. [source]
system A system message is used to specify reasoning effort, meta information like knowledge cutoff and built-in tools [source]
ollama pull hf.co/openbmb/MiniCPM5-2B-GGUF:Q4_K_M [source]
Connect Anthropic clients and tools such as Claude Code to Ollama. Ollama supports a subset of the Anthropic Messages API. [source]
Connect Anthropic clients and tools such as Claude Code to Ollama. Ollama supports a subset of the [Anthropic Messages API](https://platform.claude.com/docs/en/api/http/messages/create). [source]
Streaming is enabled by default through the REST API, but disabled by default in the SDKs. [source]
Read the answer from `choices[0].message.content`. See [OpenAI compatibility](/api/openai-compatibility) for client setup and supported features. </Tab> [source]
<Tab title="Python"> ```python theme={"system"} from ollama import chat [source]
let inThinking = false let content = '' let thinking = '' [source]
The config field references a configuration object for a container, by digest. This configuration item is a JSON blob that the runtime uses to set up the container. This new schema uses a tweaked version of this configuration to allow image content-addressability on the daemon side. [source]
Source: llama-quantize README (llama.cpp) [source]
The gpt-oss models were trained on the harmony response format for defining conversation structures, generating reasoning output and structuring function calls. If you are not using gpt-oss directly but through an API or a provider like Ollama, you will not have to be concerned about this as your inference solution will handle the formatting. If you are building your own inference solution, this guide will walk you through the prompt format. The format is designed to mimic the OpenAI Responses API, so if you have used that API before, this format should hopefully feel familiar to you. gpt-oss [source]
MLX: Leverage MLX for efficient, optimized inference of Gemma 3 QAT models on Apple Silicon. [source]
llama.cpp: Integrate easily into existing workflows thanks to native support for our GGUF-formatted QAT models. [source]
Tool calling is supported by providing a list of tools in the `tools` parameter. The model will generate a response that includes a list of tool calls. See the [Chat request (Streaming with tools)](#chat-request-streaming-with-tools) example below. [source]
Return 201 Created if the blob was successfully created, 400 Bad Request if the digest used is not expected. [source]
* Set thinking level to "low" (for GPT-OSS model). [source]
* Set thinking level to "medium" (for GPT-OSS model). [source]
* Set thinking level to "high" (for GPT-OSS model). [source]
// ValidateDigest reports whether digest names a blob. [source]
// DigestReferenceName returns the local blob digest for names that are the [source]
// the empty digest names the blobs directory [source]
path := filepath.Join(envconfig.Models(), "blobs", digest) [source]
num_ctx (int): Context window size [source]
Not where most people put it. OLLAMA_NOPRUNE is a boolean the server reads at startup, described in envconfig/config.go as "Do not prune model blobs on startup." Setting it does not make verification stricter, does not repair anything and does not affect the digest check at all. [source]
Take the digest after want, swap the colon for a hyphen, and delete that filename from the blobs folder inside your models directory — ~/.ollama/models/blobs/sha256-abc123… on macOS, /usr/share/ollama/.ollama/models/blobs/ on a standard Linux install. Blobs are shared between models, so wiping the directory throws away downloads that were never broken. [source]
Compatible models support messages, streaming, and function calling. Tool-choice controls, deferred tools, and hosted web search are not fully supported. [source]
<CodeGroup> ```python tools.py theme={"system"} import anthropic [source]
message = client.messages.create( model='qwen3-coder', max_tokens=1024, tools=[ { 'name': 'get_weather', 'description': 'Get the current weather in a location', 'input_schema': { 'type': 'object', 'properties': { 'location': { 'type': 'string', 'description': 'The city and state, e.g. San Francisco, CA' } }, 'required': ['location'] } } ], messages=[{'role': 'user', 'content': "What's the weather in San Francisco?"}] ) [source]
for block in message.content: if block.type == 'tool_use': print(f'Tool: {block.name}') print(f'Input: {block.input}') ``` [source]
```javascript tools.js theme={"system"} import Anthropic from "@anthropic-ai/sdk"; [source]
const message = await anthropic.messages.create({ model: "qwen3-coder", max_tokens: 1024, tools: [ { name: "get_weather", description: "Get the current weather in a location", input_schema: { type: "object", properties: { location: { type: "string", description: "The city and state, e.g. San Francisco, CA", }, }, required: ["location"], }, }, ], messages: [{ role: "user", content: "What's the weather in San Francisco?" }], }); [source]
for (const block of message.content) { if (block.type === "tool_use") { console.log("Tool:", block.name); console.log("Input:", block.input); } } ``` [source]
const tools = [ { type: 'function', function: { name: 'get_temperature', description: 'Get the current temperature for a city', parameters: { type: 'object', required: ['city'], properties: { city: { type: 'string', description: 'The name of the city' }, }, }, }, }, { type: 'function', function: { name: 'get_conditions', description: 'Get the current weather conditions for a city', parameters: { type: 'object', required: ['city'], properties: { city: { type: 'string', description: 'The name of the city' }, }, }, }, } ] [source]
const tools = [ { type: 'function', function: { name: 'add', description: 'Add two numbers', parameters: { type: 'object', required: ['a', 'b'], properties: { a: { type: 'integer', description: 'The first number' }, b: { type: 'integer', description: 'The second number' }, }, }, }, }, { type: 'function', function: { name: 'multiply', description: 'Multiply two numbers', parameters: { type: 'object', required: ['a', 'b'], properties: { a: { type: 'integer', description: 'The first number' }, b: { type: 'integer', description: 'The second number' }, }, }, }, }, ] [source]
const args = call.function.arguments as { a: number; b: number } console.log(`Calling ${call.function.name} with arguments`, args) const result = fn(args.a, args.b) console.log(`Result: ${result}`) messages.push({ role: 'tool', tool_name: call.function.name, content: String(result) }) } } else { break } } } [source]
This loop streams the assistant response, accumulates partial fields, passes them back together, and appends the tool results so the model can complete its answer. [source]
```bash theme={"system"} launchctl setenv OLLAMA_HOST "0.0.0.0:11434" ``` [source]
```ini theme={"system"} [Service] Environment="OLLAMA_HOST=0.0.0.0:11434" ``` [source]
PARAMETER num_ctx 4096 PARAMETER temperature 0.7 PARAMETER top_k 40 PARAMETER top_p 0.9 PARAMETER repeat_penalty 1.1 PARAMETER seed 42 [source]
TEMPLATE """{{ if .System }}<|im_start|>system {{ .System }}<|im_end|> {{ end }}{{ if .Prompt }}<|im_start|>user {{ .Prompt }}<|im_end|> {{ end }}<|im_start|>assistant """ [source]
PARAMETER num_ctx 8192 PARAMETER temperature 0.7 [source]
<|channel|>analysis<|message|>{long chain of thought}<|end|><|start|>assistant<|channel|>commentary<|message|>**Action plan**: 1. Generate an HTML file 2. Generate a JavaScript for the Node.js server 3. Start the server --- Will start executing the plan step by step<|end|><|start|>assistant<|channel|>commentary to=functions.generate_file<|constrain|>json<|message|>{"template": "basic_html", "path": "index.html"}<|call|> [source]
After all quants are built, the first quant in the list (q4_k_m) is copied to the :latest tag. This ensures a bare ollama pull without an explicit tag fetches the most portable variant. [source]
If an empty prompt is provided and the `keep_alive` parameter is set to `0`, a model will be unloaded from memory. [source]
Copy FROM yourmodel PARAMETER num_ctx 16384 [source]
You can also use the GGUF-my-repo space on Hugging Face to build your own quants without any setup. It syncs from llama.cpp main every 6 hours. [source]
FROM <some model> PARAMETER num_ctx <context size> [source]
// a manifest spells a digest with ":", a filename with "-" [source]
// DigestReference parses a bare local manifest blob reference in sha256-<hex> [source]
// ResolveDigestReference resolves a local manifest blob reference in sha256-hex, [source]
// IsDigestReferenceName reports whether n names a local manifest blob instead [source]
curl -X POST http://localhost:11434/v1/messages \ -H "Content-Type: application/json" \ -H "x-api-key: ollama" \ -H "anthropic-version: 2023-06-01" \ -d '{ "model": "qwen3-coder", "max_tokens": 1024, "messages": [{ "role": "user", "content": "Hello, how are you?" }] }' [source]
curl -X POST http://localhost:11434/v1/messages \ -H "Content-Type: application/json" \ -d '{ "model": "qwen3-coder", "max_tokens": 1024, "tools": [ { "name": "get_weather", "description": "Get the current weather in a location", "input_schema": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state" } }, "required": ["location"] } } ], "messages": [{ "role": "user", "content": "What is the weather in San Francisco?" }] }' [source]
curl http://localhost:11434/v1/messages \ -H "Content-Type: application/json" \ -d '{ "model": "claude-3-5-sonnet", "max_tokens": 1024, "messages": [ { "role": "user", "content": "Hello!" } ] }' [source]
/v1/messages/count_tokensToken counting endpoint [source]
TEMPLATE """{{ .System }} [source]
PARAMETER temperature 0.8 PARAMETER num_ctx 4096 [source]
TEMPLATE """### System: {{ .System }} [source]
PARAMETER temperature 0.7 PARAMETER num_ctx 8192 PARAMETER stop "User:" PARAMETER stop "Assistant:" [source]
TEMPLATE """[INST] {{ .System }} [source]
/openapi.yaml post /api/chat Generate the next chat message in a conversation between a user and an assistant. [source]
The Anthropic client appends `/v1/messages` to its base URL. [source]
<Tab title="cURL"> ```shell theme={"system"} curl -X POST http://localhost:11434/api/embed \ -H "Content-Type: application/json" \ -d '{ "model": "embeddinggemma", "input": "The quick brown fox jumps over the lazy dog." }' ``` </Tab> [source]
<Tabs> <Tab title="cURL"> ```shell theme={"system"} curl -X POST http://localhost:11434/api/embed \ -H "Content-Type: application/json" \ -d '{ "model": "embeddinggemma", "input": [ "First sentence", "Second sentence", "Third sentence" ] }' ``` </Tab> [source]
Type `/model` inside Codex to switch between your installed, tool-capable Ollama models without restarting. The starting model remains selected, and each model keeps its own context window, vision, and thinking settings. Pull additional models with `ollama pull <model>` before launching to include them in the picker. [source]
<Tab title="OpenAI Responses"> ```shell theme={"system"} curl http://localhost:11434/v1/responses \ -H "Content-Type: application/json" \ -d '{ "model": "gemma4:e2b", "input": "Say hello in one sentence." }' ``` [source]
QWEN3_4B_THINKING("qwen3:4b-thinking"), [source]
* **AMD GPU**: `ollama-windows-amd64-rocm.zip` * **MLX (CUDA)**: `ollama-windows-amd64-mlx.zip` [source]
Response fields: chat returns reasoning in `message.thinking` and the answer in `message.content`; generate returns `thinking` and `response`. [source]
* `OLLAMA_KV_CACHE_TYPE` - The quantization type for the K/V cache. Default is `f16`. [source]
How much the cache quantization impacts the model's response quality will depend on the model and the task. Models that have a high GQA count (e.g. Qwen2) may see a larger impact on precision from quantization than models with a low GQA count. [source]
You may need to experiment with different quantization types to find the best balance between memory usage and quality. [source]
The reasoning output and answer use separate fields. Chat returns `message.thinking` and `message.content`. Generate returns `thinking` and `response`. [source]
print('Thinking:\n', response.message.thinking) print('Answer:\n', response.message.content) ``` </Tab> [source]
console.log('Thinking:\n', response.message.thinking) console.log('Answer:\n', response.message.content) ``` </Tab> </Tabs> [source]
本記事はOllama公式ドキュメント（Modelfile Reference）の仕様を一次ソースとして、全インストラクションの構文・全パラメータの既定値・ビルド手順を整理する（2026-07-19確認）。Ollama自体の導入やモデル選定についてはOllama導入ガイドとOllamaモデル選定ガイドを参照してほしい。 [source]
先頭のコメント行：# Modelfile generated by "ollama show" に続き、# To build a new Modelfile based on this one, replace the FROM line with: # FROM llama3.2:latest と書かれている。出力されたFROMはblobの実体パスを指しているため、派生モデルを作るときはこの行をモデル名に書き換えるのが公式の指示だ [source]
FROM ./models/your-model-q4_k_m.gguf [source]
ビルドはollama create <名前> -f ./Modelfile、実行はollama run <名前> [source]
// { host } "/" { namespace } "/" { model } ":" { tag } "@" { digest } [source]
// { namespace } "/" { model } ":" { tag } "@" { digest } [source]
// { model } ":" { tag } "@" { digest } [source]
MLX and llama.cpp updates [source]
changed the title Create safetensors models through a hybrid local/remote pipeline. [source]
We should just deprecate this for now since LoRAs aren't supported. [source]
I added the local GGUF as a discrete commit in case we want to drop it. [source]
- `quantize` (optional): quantize a non-quantized (e.g. float16) model [source]
- `quantize` (optional): quantize model weights during import [source]
Importing a model from Safetensors weights [source]
For a split GGUF model, keep the original split filenames and use a wildcard that matches every shard. [source]
FROM /path/to/model-*.gguf [source]
The format of the `Modelfile`: [source]
An example of a `Modelfile` creating a mario blueprint: [source]
The `LICENSE` instruction allows you to specify the legal license under which the model used with this Modelfile is shared or distributed. [source]
Related: How to generate text with the Ollama API [source]
Related: How to run a chat request with the Ollama API [source]
#Quantization: what you trade for memory [source]
Weights only, in GB, according to quantization (calculated using the formula above) [source]
#What quantization really does to quality [source]
Explicitly choose a quantization in Ollama⧉ Copy [source]
Quantize the KV cache: save VRAM [source]
Claude Code with Anthropic API compatibility [source]
Claude Code is Anthropic’s agentic coding tool that lives in your terminal. With Anthropic API support, you can now use Claude Code with any Ollama model. [source]
Existing applications using the Anthropic SDK can connect to Ollama by changing the base URL. See the Anthropic compatibility documentation for details. [source]
For a complete list of supported features, see the Anthropic compatibility documentation. [source]
GitHub issues are welcome! For more information, see the OpenAI compatibility docs. [source]
LoRA is orthogonal to other parameter-efficient methods and can be combined with many of them. [source]
Performance of models finetuned using LoRA is comparable to the performance of fully finetuned models. [source]
r (int) — Lora attention dimension (the “rank”). [source]
Quantization and Deployment for Finetuned Models [source]
Quantization and Deployment for Finetuned Models [source]
Produce an intermediate F16 GGUF — a lossless 16-bit GGUF is generated first as the input for imatrix computation [source]
Compute activation statistics — llama-imatrix runs inference on the calibration text using the F16 GGUF, recording how much each weight matrix contributes to the model’s outputs across the calibration corpus; the result is a .imatrix.dat file [source]
The GPU-served NF4 model serves as the reference; all GGUF variants are compared against it. [source]
For complex incident analysis specifically, only the NF4/GPU variant is competitive. If complex incidents are a priority deployment target, do not use quantized variants without further fine-tuning on longer DAGs. [source]
How to create the model in Ollama with a Modelfile [source]
What happens if you import the safetensors directly [source]
Can I delete the GGUF after importing it? [source]
--experimental Turns on experimental safetensors creation Hidden flag that does nothing [source]
llama-server as an OpenAI-compatible API in Docker, tested with the official SDKs [source]
Understanding performance, precision, and quantization [source]
The impact of int4 quantization is dramatic. Look at the VRAM (GPU memory) required just to load the model weights: [source]
Example of streaming responses with thinking [source]
Save it as a file (e.g. Modelfile) [source]
Convert the original model to GGUF format. [source]
If the model supports multimodal inputs (images or audio), you also need to convert and quantize the multimodal encoders and projectors. [source]
Various quantization methods are described later in this document. [source]
--pure disable k-quant mixtures and quantizes all tensors to the same type [source]
--tensor-type quantize specific tensor(s) to specific quant types. Supports regex syntax. May be specified multiple times. [source]
--override-kv option to override model metadata by key in the quantized model. May be specified multiple times. [source]
./build/bin/llama cli -m ./gemma-4-E2B-it-Q4_K_M.gguf --mmproj ./mmproj-gemma-4-E2B-it-Q8_0.gguf --image <input_image> --prompt "Describe this image" [source]
Several quantization methods are supported. They differ in the resulting model disk size and inference speed. For example, [source]
quantization tuning, another one, and another one [source]
.metadata("thinking", ollamaResponse.message().thinking()) [source]
Short: -fPath to the Modelfile. Defaults to Modelfile in the current directory. [source]
Keep your Modelfile in version control and document what each version changes. [source]
Ollama Mastery is 15 chapters end to end — install, model choice, Modelfiles, GPU offload, the API, and the 20 errors that actually happen. Plus 24 more courses. [source]
The OpenAI API does not have a way of setting the context size for a model. If you need to change the context size, create a `Modelfile` which looks like: [source]
* Define schemas with Pydantic (Python) or Zod (JavaScript) so they can be reused for validation. * Lower the temperature (e.g., set it to `0`) for more deterministic completions. * Structured outputs work through the OpenAI-compatible API via `response_format` [source]
Provide an `images` array. SDKs accept file paths, URLs or raw bytes while the REST API expects base64-encoded image data. [source]
Ollama can be integrated into most of the tools available either through direct integration of Ollama's API, Python / JavaScript libraries, OpenAI compatible API, and MCP server integration. [source]
<Card title="Build an application" icon="code" href="/quickstart#build-an-application"> Call cloud models with Ollama, OpenAI, or Anthropic APIs. </Card> </CardGroup> [source]
Ollama connects Claude Code to local and cloud models through its Anthropic-compatible API. [source]
Choose a [cloud model](https://ollama.com/search?c=cloud) that supports tools. Basic chat and file edits work with compatible models. Hosted WebSearch and advanced tool controls are not fully supported. See [Anthropic compatibility](/api/anthropic-compatibility). [source]
1. Create an [API key](https://ollama.com/settings/keys) from ollama.com 2. Click on `Use custom base URL` and set it to `https://ollama.com` 3. Enter your **Ollama API Key** 4. Select a model from the list [source]
Copilot CLI connects to Ollama using the OpenAI-compatible API via environment variables. [source]
3. Set the API base URL to the Ollama OpenAI-compatible endpoint: [source]
Pool connects to Ollama using the OpenAI-compatible API via environment variables. [source]
1. Create an [API key](https://ollama.com/settings/keys) from ollama.com 2. Select **Internet Hosted** and enter URL as `https://ollama.com` 3. Enter your **Ollama API Key** and click **Add** [source]
Responses requests are stateless: include conversation history in each request. `previous_response_id` and `conversation` aren't supported. See [Responses compatibility](/api/openai-compatibility#responses-api). </Tab> [source]
Read text blocks from `content`. Direct cloud requests use bearer authentication. See [Anthropic compatibility](/api/anthropic-compatibility) for client setup and supported features. </Tab> </Tabs> [source]
OpenAI and Anthropic compatibility each cover a subset of the original API. See [Cloud](/cloud) for models and usage limits. </Tab> [source]
Read text blocks from `content`. See [Anthropic compatibility](/api/anthropic-compatibility) for client setup and supported features. </Tab> </Tabs> </Tab> </Tabs> [source]
This includes importing foundation models as well as any fine tuned models which have been _fused_ with a foundation model. [source]
this.think = ThinkOption.ThinkLevel.HIGH; [source]
4. Edit or create a new variable for your user account for `OLLAMA_HOST`, `OLLAMA_MODELS`, etc. [source]
Ollama binds 127.0.0.1 port 11434 by default. Change the bind address with the `OLLAMA_HOST` environment variable. [source]
If a different directory needs to be used, set the environment variable `OLLAMA_MODELS` to the chosen directory. [source]
【最小構成】Modelfileを書いて ollama create でビルドする手順 [source]
num_ctx 次のトークン生成に使うコンテキストウィンドウのサイズ int 2048 [source]
repeat_last_n 繰り返し防止のためにどこまで遡るか（0=無効、-1=num_ctx） int 64 [source]
置き場所に決まりはありません。任意のディレクトリにModelfileとして保存し、ollama create <モデル名> -f ./Modelfileのように-fでパスを渡します。ただしFROMでGGUFやSafetensorsを相対パス指定する場合、その基準はModelfileの場所になります。 [source]
できます。ollama show --modelfile llama3.2のように実行すると、そのモデルのFROM・TEMPLATE・PARAMETERの定義が出力されます。TEMPLATEやstopはモデル固有のため、自作する前に元の定義を確認するのが確実です。 [source]
// - The default host is ("registry.ollama.ai") [source]
Verify the client waits until the final done chunk arrives. $ python3 - <<'PY' last = {"done": True, "done_reason": "stop"} print(last["done"]) PY True [source]
export ANTHROPIC_AUTH_TOKEN=ollama export ANTHROPIC_BASE_URL=http://localhost:11434 [source]
config_list = [ { "model": "codellama", "base_url": "http://localhost:11434/v1", "api_key": "ollama", } ] [source]
For endpoints that return usage metrics, the response body will include the usage fields. For example, a non-streaming call to `/api/generate` may return the following response: [source]
> Note: this endpoint has been superseded by `/api/embed` [source]
TEMPLATEThe full prompt template to be sent to the model. [source]
repeat_last_nSets how far back for the model to look back to prevent repetition. (Default: 64, 0 = disabled, -1 = num_ctx)intrepeat_last_n 64 [source]
Why the num_ctx you already set did nothing [source]
prompt_eval_count in the API response. If your prompt is clearly 9,000 tokens and this comes back near your window size, the rest was discarded. [source]
The fix you should make even after fixing num_ctx [source]
I set num_ctx and nothing changed. Why? [source]
Tags:ollama num_ctxollama context windowollama truncatesagent ignores soul.mdollama context lengthOLLAMA_CONTEXT_LENGTHollama num_ctx not workingollama v1 endpoint num_ctxhermes ollama context [source]
Choose a cloud model. The cloud API does not support stateful Responses, built-in web search through /v1/responses, or custom/freeform tool-call replay. [source]
curl -X POST http://localhost:11434/v1/responses \ -H "Content-Type: application/json" \ -d '{ "model": "qwen3:8b", "input": "Write a short poem about the color blue" }' [source]
/v1/chat/completions with vision example [source]
spring-ai-docs/src/main/antora/modules/ROOT/pages/api/chat/ollama-chat.adoc [source]
@JsonProperty("keep_alive") String keepAlive, [source]
private static final List<String> NON_SUPPORTED_FIELDS = List.of("model", "format", "keep_alive", "truncate"); [source]
private static final List<String> NON_SUPPORTED_FIELDS = List.of("model", "format", "keep_alive", "truncate", [source]
defaultPublicHost = "registry.ollama.ai" [source]
/api/generate Generates a response for the provided prompt [source]
Chat template that formats prompts. Uses Go template syntax. [source]
As a work-around, you could have the clients use the OpenAI API endpoint, or use a proxy to remove num_ctx from the request. [source]
indeed, I patched ollama sources to ignore num_ctx, but it would be nice, if supported directly by ollama. [source]
OLLAMA_MODELS Not Working? Fix It by Platform [source]
/openapi.yaml get /api/version Retrieve the version of the Ollama [source]
The local API at `http://localhost:11434` does not require authentication. [source]
/openapi.yaml post /api/embed Creates vector embeddings representing the input text [source]
/openapi.yaml post /api/generate Generates a response for the provided prompt [source]
Choose a [cloud model](https://ollama.com/search?c=cloud). The cloud API does not support stateful Responses, built-in web search through `/v1/responses`, or custom/freeform tool-call replay. [source]
/openapi.yaml get /api/ps Retrieve a list of models that are currently running [source]
/openapi.yaml get /api/tags Fetch a list of models and their details [source]
<Note> The `/api/embed` endpoint returns L2‑normalized (unit‑length) vectors. </Note> [source]
Select Ollama as the provider, use `http://localhost:11434` as the base URL if prompted, and choose a model such as `qwen3.5` or `kimi-k2.6:cloud`. [source]
2. Find **Ollama**, click **Configure** 3. Confirm **API Host** is `http://localhost:11434` and click Submit [source]
1. **Install** — If Hermes isn't installed, Ollama prompts to install the Hermes command-line agent 2. **Model** — Pick a model from the selector (local or cloud). See currently-supported models [here](https://ollama.com/search). 3. **Onboarding** — Ollama configures the Ollama provider, points Hermes at `http://127.0.0.1:11434/v1`, and sets your model as the primary 4. **Gateway** — Optionally connects a messaging platform (Telegram, Discord, Slack, WhatsApp, Signal, Email) and launches the Hermes chat [source]
``` API base URL [e.g. https://api.example.com/v1]: http://127.0.0.1:11434/v1 ``` [source]
``` Verified endpoint via http://127.0.0.1:11434/v1/models (1 model(s) visible) Detected model: kimi-k2.5:cloud Use this model? [Y/n]: ``` [source]
3. Under **Third Party AI Providers**, choose **Ollama** 4. Confirm the **Host URL** is `http://localhost:11434`, then click **Ok** 5. Once connected, select a model under **Local models by Ollama** [source]
1. In marimo, go to the user settings and go to the AI tab. From here you can find and configure Ollama as an AI provider. For local use you would typically point the base url to `http://localhost:11434/v1`. [source]
3. Confirm Base URL is set to `http://localhost:11434` if running locally or `http://host.docker.internal:11434` if running through docker and click **Save** [source]
The extension discovers models from `http://127.0.0.1:11434` by default. [source]
2. Select **Locally Hosted**, enter port **11434** and click **Add** [source]
2. Under **LLM Providers**, choose **Ollama** 3. Confirm the **Host URL** is `http://localhost:11434`, then click **Connect** 4. Once connected, select a model under **Ollama** [source]
Ollama runs as a native Windows application, including NVIDIA and AMD Radeon GPU support. After installing Ollama for Windows, Ollama will run in the background and the `ollama` command line is available in `cmd`, `powershell` or your favorite terminal application. As usual the Ollama [API](/api) will be served on `http://localhost:11434`. [source]
Thinking-capable models emit a `thinking` field that separates their reasoning trace from the final answer. [source]
The response includes a top-level `thinking` object: [source]
The llama-quantize README lists, for Llama 3.1 8B, the number of bits per weight and the size of each format. The Ollama library shows the size of the distributed files for the same model. The two sources broadly agree and provide an estimate applicable to other models in the same family. [source]
Sharp drop in reasoning: avoid without a memory-related reason. [source]
Choose the largest model that fits in Q4_K_M [source]
If VRAM remains after reserving space for the context, move to Q5_K_M and then Q6_K. The gain is smaller than moving up to the next size. [source]
The reasoning effort — As specified on the levels high, medium, low [source]
Handling reasoning output in subsequent sampling [source]
Evaluation basis: performance numbers in this document were measured on the finetuned summarization model (47 held-out incidents, judge: gpt-oss-120b). The GPU-served reference is serve_model.py with --quant 4bit (bitsandbytes NF4), not true fp16. The conversion and publication methodology applies to any finetuned model in this pipeline. [source]
cd unsloth-scripts/ python3 convert_to_gguf.py \ --model /path/to/qwen_finetuned_merged_16bit \ --quant q5_k_m \ --output ./gguf_q5_k_m/ [source]
./publish_to_ollama.sh --quant q5_k_m [source]
q5_k_m — good middle ground on both tasks [source]
With 4 threads, the Q4_K_M generation medians ranged from 32 to 70 tokens/s depending on the tool and the load. The default thread count was the most serious problem in the test. Without num_thread, Ollama started its internal llama-server with 18 threads, one per core. [source]
Gemma 3 QAT Models: Bringing state-of-the-Art AI to consumer GPUs [source]
Last month, we launched Gemma 3, our latest generation of open models. Delivering state-of-the-art performance, Gemma 3 quickly established itself as a leading model capable of running on a single high-end GPU like the NVIDIA H100 using its native BFloat16 (BF16) precision. [source]
The chart above shows the performance (Elo score) of recently released large language models. Higher bars mean better performance in comparisons as rated by humans viewing side-by-side responses from two anonymous models. Below each bar, we indicate the estimated number of NVIDIA H100 GPUs needed to run that model using the BF16 data type. [source]
Performance Meets Accessibility with Quantization-Aware Training in Gemma 3 [source]
Ollama: Get running quickly – all our Gemma 3 QAT models are natively supported starting today with a simple command. [source]
LM Studio: Easily download and run Gemma 3 QAT models on your desktop via its user-friendly interface. [source]
Bringing state-of-the-art AI performance to accessible hardware is a key step in democratizing AI development. With Gemma 3 models, optimized through QAT, you can now leverage cutting-edge capabilities on your own desktop or laptop. [source]
Ollama now has the ability to enable or disable thinking. This gives users the flexibility to choose the model’s thinking behavior for different applications and use cases. [source]
more will be added under thinking models. [source]
In the CLI, thinking is enabled by default. [source]
In the CLI, thinking is disabled using /set nothink followed by the prompt. [source]
For scripting, a --hidethinking command is available. This helps users who want to use thinking models but simply want to see the answer. [source]
Example using Ollama’s chat API with thinking enabled [source]
Background information on llama-quantize [source]
Reasoning/thinking control (for thinking models) [source]
reasoning_effort (model-defined names and compatibility aliases) [source]
Reasoning summaries (for thinking models) [source]
feat(ollama): add thinking mode support for reasoning models #4821 [source]
feat(ollama): add thinking mode support for reasoning models [source]
* Enable thinking mode for the model. [source]
* Disable thinking mode for the model. [source]
* <li>Standard models (e.g., qwen2.5:*, llama3.2) do not enable thinking by default.</li> [source]
* Enable thinking mode for the model. The model will include its reasoning [source]
* Qwen3 4B with thinking support. This variant auto-enables thinking by default in [source]
* Ollama 0.12+, providing separate reasoning traces in the response. [source]
content (text, tool_use, thinking blocks) [source]
q4_K_M~4 bits/weightGoodFastRecommended for most use cases [source]
q5_K_M~5 bits/weightBetterMediumHigher quality, larger [source]
q8_0~8 bits/weightBestSlowHighest quality, largest [source]
Browse the [model library](https://ollama.com/search) for coding, vision, embeddings, and reasoning models. [source]
<div> <div>Thinking</div> <div>Use thinking controls with compatible models</div> </div> </div> </div> </div> [source]

Related concepts

MESSAGE — is a related of Ollama core platform
SYSTEM — is a related of Ollama core platform
localhost:11434 — is a part of Ollama core platform
stream — is a part of Ollama core platform
PARAMETER — is a part of Ollama core platform
q4_K_M — is a instance of Ollama core platform
blob — is a part of Ollama core platform
llama.cpp — is a related of Ollama core platform
tool_calls — is a part of Ollama core platform
sha256 digest — is a part of Ollama core platform
TEMPLATE — is a part of Ollama core platform
ollama create — is a part of Ollama core platform
num_ctx — is a part of Ollama core platform
ADAPTER — is a part of Ollama core platform
fp16 — is a instance of Ollama core platform
/api/chat — is a part of Ollama core platform
MLX — is a related of Ollama core platform
think — is a part of Ollama core platform
manifest — is a part of Ollama core platform
/api/generate — is a part of Ollama core platform

Children

Adapter architecture tag mismatch (frontier)
Choosing quantization for VRAM budget (frontier)
LoRA adapters in Ollama via ADAPTER
MESSAGE few-shot seeding (frontier)
MLX safetensors create pipeline (frontier)
Merging LoRA before GGUF import (frontier)
Ollama /api/show capability discovery (frontier)
Ollama Modelfile authoring: FROM and PARAMETER and TEMPLATE and SYSTEM and ADAPTER and MESSAGE and LICENSE
Ollama OpenAI and Anthropic compatibility endpoints
Ollama REST API: generate and chat with streaming NDJSON and errors and usage
Ollama cloud models via /v1 (frontier)
Ollama context length defaults and num_ctx precedence (frontier)
Ollama importing models: GGUF and safetensors and quantize on create
Ollama keep_alive and model load/unload via empty requests (frontier)
Ollama model quantization tags and library variants
Ollama model registry and library tags and on-disk blob and manifest storage
Ollama structured outputs via format (frontier)
Ollama thinking capability and reasoning controls
Ollama thinking via OpenAI-compatible endpoints (frontier)
Ollama tool calling over /api/chat streaming (frontier)
QAT builds in Ollama library (frontier)
Split GGUF shard import (frontier)
Writing Go templates for chat and tool-calling models (frontier)
gpt-oss reasoning levels (frontier)
llama.cpp convert_hf_to_gguf and llama-quantize workflow (frontier)
llama.cpp convert_lora_to_gguf.py (frontier)
manifests-v2 canonical host migration (frontier)
ollama launch claude integration (frontier)
ollama pull Range/part download and digest verification (frontier)

Frontier under this node: Adapter architecture tag mismatch, Choosing quantization for VRAM budget, MESSAGE few-shot seeding, MLX safetensors create pipeline, Merging LoRA before GGUF import, Ollama /api/show capability discovery, Ollama cloud models via /v1, Ollama context length defaults and num_ctx precedence, Ollama keep_alive and model load/unload via empty requests, Ollama structured outputs via format, Ollama thinking via OpenAI-compatible endpoints, Ollama tool calling over /api/chat streaming, QAT builds in Ollama library, Split GGUF shard import, Writing Go templates for chat and tool-calling models, gpt-oss reasoning levels, llama.cpp convert_hf_to_gguf and llama-quantize workflow, llama.cpp convert_lora_to_gguf.py, manifests-v2 canonical host migration, ollama launch claude integration, ollama pull Range/part download and digest verification

← the whole tree · 3D view· how to read this page