Ollama server ops, scheduling, GPU, sizing and desktop

Parent: Ollama · Published reference · snapshot 2026-10-14 · skill ollama-expert/references/ollama-server-ops-scheduling-gpu-sizing-and-desktop.md

↓ Facts as markdown all context files

Ollama server operations covers how the Ollama server schedules requests (OLLAMA_NUM_PARALLEL, OLLAMA_MAX_LOADED_MODELS, OLLAMA_MAX_QUEUE, keep_alive eviction), selects GPU backends (CUDA, ROCm, Vulkan), estimates memory and places layers across GPUs (OLLAMA_SCHED_SPREAD), configures flash attention and KV cache type, sizes RAM and VRAM, installs and runs as a service on Linux and Windows, and exposes desktop-app settings and vision input; it is compiled from 34 source files across docs.ollama.com, Ollama source code, blog posts and community guides.

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Definitions

Cache size scales with num_ctx and is not changed by weight quantization. For llama3.1:8b at f16 the cache is about 128 KiB per token: 4 GiB at 32,768 tokens, 8 GiB at 65,536; q8_0 at 32,768 tokens is about 2.13 GiB. Sliding-window or shared-KV models (Gemma family) break this formula, so sum the 'llama_kv_cache: size' log lines instead. [source]
This typically means the system's AMD GPU driver is too old. Ollama bundles ROCm 7 linux libraries which require a compatible ROCm 7 kernel driver. If the system is running an older driver (ROCm 6.x or earlier), GPU initialization will hang during device discovery and eventually time out, causing Ollama to fall back to CPU. [source]
Capping the context is one of two fixes. The other keeps the long context and shrinks the cache instead. On the RTX 4090, Qwen3.8 27B at a 128K context spilled with the default f16 cache and dropped to 16.4 tok/s. With OLLAMA_KV_CACHE_TYPE=q4_0 and flash attention active, the identical context stayed 100 percent on the GPU. The KV cache type selector in the calculator reproduces both outcomes. [source]
OLLAMA_SCHED_SPREAD is a boolean env var described in source as 'Always schedule model across all GPUs', overriding the prefer-single-GPU default. [source]
Rost Glukhov is a polyglot developer, has a special interest in web, mobile, backend, AI, devops and other areas. For more details have a look at the About page. For partnership and sponsored content inquiries, see Partnerships. [source]
Parallel execution in Ollama means handling multiple requests at the same time, either for the same model or for different models, depending on available memory and configuration. [source]
* If you are using a container, is the container runtime working? Try `docker run --gpus all ubuntu nvidia-smi` - if this doesn't work, Ollama won't be able to see your NVIDIA GPU. * Is the uvm driver loaded? `sudo nvidia-modprobe -u` * Try reloading the nvidia\_uvm driver - `sudo rmmod nvidia_uvm` then `sudo modprobe nvidia_uvm` * Try rebooting * Make sure you're running the latest nvidia drivers [source]
The KV cache type is the lever that acts on the right-hand bars. Weight quantization does nothing to them. [source]
Vulkan itself is a cross-platform graphics API that works across a wide range of GPU manufacturers, including Intel and older AMD cards outside the ROCm compatibility list. [source]
This is the single most out-of-date piece of Ollama advice in circulation, and this page carried it too until this update. The change landed in commit fdb1094 on 2025-10-01, which swapped the variable from a plain boolean to a three-state one and deleted the branch that required you to opt in. Ollama's own FAQ was not corrected until 2026-07-03, in PR #16994, which shipped in v0.31.2 on 2026-07-06 and removed the sentence "To enable Flash Attention, set the OLLAMA_FLASH_ATTENTION environment variable to 1". For nine months the official documentation told people to set a variable that had alrea [source]
<Note> Currently this is a global option - meaning all models will run with the specified quantization type. </Note> [source]
OLLAMA_MAX_LOADED_MODELS is how many different models stay resident at once. The default is 0, which means Ollama picks: three models per GPU, and three on a machine with no GPU. [source]
Changing the Ollama port is one line in a systemd drop-in. Everything that follows, the firewall scope, the reverse proxy and the blocked management endpoints, is what keeps an unauthenticated inference API from becoming a public one. Verify the bound address with ss -tlnp after every change, since a drop-in that never took effect looks identical to a working configuration until a client tries to connect. [source]
Running Ollama on Windows as a real system service is not provided by the installer: the docs point to the standalone ollama-windows-amd64.zip (CLI plus NVIDIA libs; add ollama-windows-amd64-rocm.zip for AMD) run via `ollama serve` under a wrapper such as NSSM. [source]
GPU acceleration is not available for Docker Desktop in macOS due to the lack of GPU passthrough and emulation. [source]
// loadedModel is a point-in-time snapshot of a loaded runner's state, safe to // use without holding any scheduler locks. type loadedModel struct { model *Model runner string size int64 sizeVRAM int64 contextLength int expiresAt time.Time } [source]
What is the OLLAMA_KV_CACHE_TYPE variable? [source]

Structure and components

Vendor-reported benchmark: gemma3:12b at 128k context on one RTX 4090 went from 48/49 to 49/49 GPU layers and 52.02 to 85.54 tok/s; mistral-small3.2 with image input on two RTX 4090s went from 40/41 to 41/41 layers. The post gives no mechanism detail for multi-GPU gains and does not mention SCHED_SPREAD. [source]
Multi-GPU does not pool into one fast GPU: layers/tensors are split across devices with cross-device transfer overhead, so its main value is fitting larger models, not linear speedup; forcing a split of a model that fits one GPU can be slower. [source]
The num_gpu option is a count of layers to offload, not a number of GPUs. [source]
No. It lets more requests run at the same time, and each of those requests is slower than it would be running alone, because they share one GPU. It also multiplies the KV cache, since Ollama starts the runner with a total context of your context length times the slot count. If the result no longer fits in VRAM, Ollama pushes layers onto the CPU and every request slows down, including a single one with no competition. Check ollama ps after the change and confirm the PROCESSOR column still reads 100% GPU. [source]
// Pre-flight check: estimate whether the model fits in remaining memory. // llama-server auto-detects layers based on available VRAM, so if // we predict it won't fit, evict before spawning. if requireFull && !explicitPartialGPUOffload(launchOpts, f) && len(s.loaded) > 0 && len(loadGpus) > 0 { freeMemory, gpuFreeMemory, systemLimited := availableMemoryForPlacement(systemInfo, loadGpus, launchOpts) // Use 80% of free memory as threshold to leave headroom. if predictedForLoad > freeMemory*80/100 { slog.Info("llama-server model predicted to exceed available memory, evicting", "predicted", format [source]
Because the KV cache is proportional to the context length while the weights are fixed. For llama3.1:8b the cache costs 128 KiB per token at f16, which is 4 GiB at 32768 tokens and 8 GiB at 65536. Dropping the weights from q8_0 to q4_K_M saves about 3.4 GiB, so one doubling of the context undoes the whole saving. For a model where every layer uses full attention and keeps its own cache, as llama3.1 does, run ollama show <model> --verbose and multiply 2 x layers x KV heads x head dimension x 2 bytes. If that output lists sliding_window or shared_kv_layers, the multiplication does not hold, so r [source]
Total VRAM is weights plus KV cache plus runtime overhead. KV bytes per token = 2 x attention layers x KV heads x head dim x bytes per element (f16 2, q8_0 1.06, q4_0 0.56). Example: Llama 3.1 8B uses 128 KiB per token, about 4.3 GB at 32K context. Runtime overhead is about 0.7 GB at short context, paid per GPU. [source]
The first three come from the config.json of the model as num_hidden_layers, num_key_value_heads and head_dim, with layer_types telling you which layers are full attention on hybrid models. Where head_dim is missing, as in the Qwen2.5 configs, divide hidden_size by num_attention_heads. The fourth is the cache type. Llama 3.3 70B has 80 layers with 8 key-value heads of dimension 128, so 2 × 80 × 8 × 128 × 2 bytes is 320 KiB per token. Grouped-query attention is what keeps that small: with a separate key and value for each of its 64 query heads, the same model would need 2.5 MiB per token. The c [source]
// Some architectures are not safe with num_parallel > 1. // ref: https://github.com/ollama/ollama/issues/4165 if slices.Contains([]string{"mllama", "qwen3vl", "qwen3vlmoe", "qwen35", "qwen35moe", "qwen3next", "lfm2", "lfm2moe", "nemotron_h", "nemotron_h_moe", "nemotron_h_omni"}, req.model.Config.ModelFamily) && numParallel != 1 { numParallel = 1 slog.Warn("model architecture does not currently support parallel requests", "architecture", req.model.Config.ModelFamily) } [source]
Yes. Two 24 GB cards give 48 GB of combined VRAM, which holds Llama 3.3 70B at Q4_K_M with a 16K context and about 2 GB to spare, and it is often cheaper than one 48 GB card. Each card carries its own runtime overhead, which the calculator counts per GPU. Ollama and llama.cpp both split the layers across GPUs automatically; in llama.cpp that is the default --split-mode layer, and --tensor-split only changes the ratio. NVLink is not required; a pair of cards on ordinary PCIe slots works, adding a small inter-GPU latency. The catch is that VRAM scales but speed does not. Splitting a 70B across t [source]
On a 24 GB card the auto-sized 32k default context for a 32B Q4 model can push total memory past VRAM, spilling layers to CPU and cutting throughput to about a tenth. Fix: cap OLLAMA_CONTEXT_LENGTH or num_ctx. [source]
Two models of the same size can need wildly different VRAM at the same context, and parameter count gives no hint of it. The KV cache per token depends on how many layers use full attention and how many key-value heads they keep. Qwen3 32B stores 256 KiB for every token of context. Qwen3.8 27B stores 64 KiB, because only 16 of its 64 layers are full attention and the rest are Gated DeltaNet layers with a fixed-size state. Nemotron 3.5 Lightning stores 6 KiB. The figures below are computed from the config.json of each model at the default f16 cache type: [source]
// Free memory reporting on GPUs can lag for a while even after the runner // exits, so we have to keep checking until we see the available memory recover, // otherwise subsequent model loads will get far less layers loaded or worse // case, may completely fall back to CPU mode. // This routine must be called before the runner unloads so it can establish // a before and after GPU memory allocation. The returned channel // will be notified when we're done waiting, or have timed out and should // proceed anyway func (s *Scheduler) waitForVRAMRecovery(runner *runnerRef, runners []ml.FilteredRunn [source]
No. Ollama has no environment variable for KV cache offload and does not expose the underlying flag. It offloads model layers and the KV cache to the GPU together. [source]
Part of our series on running Ollama on Mac. See also: Ollama KV cache quantization, the Ollama CORS fix, and running multiple Ollama models on Mac. [source]
Additional GPU support on Windows and Linux is provided via [Vulkan](https://www.vulkan.org/). Vulkan is enabled by default when the backend is installed. On Windows most GPU vendors drivers come bundled with Vulkan support and require no additional setup steps. Most Linux distributions require installing additional components, and you may have multiple options for Vulkan drivers between Mesa and GPU Vendor specific packages [source]
No, not on a single 24 GB card in a usable way. A 70B at Q4_K_M needs 42.5 GB before context, and a 24 GB card is short by almost half. The runtime will start it by offloading about half the layers to the CPU, and throughput drops to one or two tokens per second, slower than reading speed. The real options for a 70B are a single 48 GB card such as the RTX 6000 Ada, two 24 GB cards splitting the model, or a unified-memory machine with 64 GB or more. A single 24 GB card is the right tool for a 27B to 32B model, not a 70B. [source]
For GPU inference a model should fit fully in VRAM; ollama ps shows the Processor column (100% GPU, 100% CPU, or a CPU/GPU split such as 48%/52%). A split means some layers run on CPU from system RAM and is much slower. [source]
• Memory estimations (e.g. how many layers are offloaded to each GPU). [source]
KV stands for key and value. For every token in the context, each attention layer stores one key vector and one value vector, so later tokens can attend to it. That store is the KV cache. It is a different thing from the prompt cache that lets a repeated prefix skip prefill; the KV cache and the prompt cache are often confused, and only the first one costs you memory in proportion to num_ctx. [source]
40⁄41 layers loaded on GPU 41⁄41 layers loaded on GPU + vision model [source]
More than inference, and it depends on the method. Full fine-tuning loads the weights, the gradients, and the optimizer states, which pushes a 7B well past 60 GB and often beyond 100 GB, and puts a 70B firmly in multi-GPU territory. Parameter-efficient methods change that completely. QLoRA quantizes the base model to 4-bit and trains only small adapter layers on top, which brings a 7B fine-tune under 16 GB and a 70B fine-tune to roughly 48 GB, the same single 48 GB card that runs a 70B at inference. The Unsloth QLoRA walkthrough runs one end to end on a single card. [source]
Old New 52.02 tokens/s token generation speed 85.54 tokens/s token generation speed 19.9GiB of VRAM 21.4GiB of VRAM 48⁄49 layers loaded on GPU 49⁄49 layers loaded on GPU [source]
That matches the arithmetic above exactly: 128 KiB per token times 32768 tokens. K (f16) and V (f16) are the parts you are about to shrink. Note the Flash Attention line too. With --flash-attn auto, llama.cpp probes the backend and logs whether it ended up enabled, and the next step depends on that answer. [source]
Copy llama_kv_cache: size = 2176.00 MiB ( 32768 cells, 32 layers, 1/1 seqs), K (q8_0): 1088.00 MiB, V (q8_0): 1088.00 MiB [source]
Today, ggml/llama.cpp offers first-class support for text-only models. For multimodal systems, however, the text decoder and vision encoder are split into separate models and executed independently. Passing image embeddings from the vision model into the text model therefore demands model-specific logic in the orchestration layer that can break specific model implementations. [source]
Memory is the easy half. The weights are a file size, and the KV cache is exact arithmetic once the layer layout is known. The checks below compare the calculator against memory actually allocated on real hardware: [source]
Offloading also sets a floor on system RAM. The layers that spill out of VRAM live in main memory, so size system RAM to hold the offloaded portion plus headroom for the OS. GGUF memory-maps the file, so only the offloaded layers stay resident, and fast NVMe speeds the initial load but does nothing for generation once the model is mapped. [source]
48⁄49 layers loaded on GPU 49⁄49 layers loaded on GPU [source]
System RAM becomes important when the full workload doesn’t fit on the GPU, since Ollama runs the part that doesn’t fit in VRAM on the CPU instead. [source]

How it works

func AsMap() map[string]EnvVar { ret := map[string]EnvVar{ "OLLAMA_DEBUG": {"OLLAMA_DEBUG", LogLevel(), "Show additional debug information (e.g. OLLAMA_DEBUG=1)"}, "OLLAMA_DEBUG_LOG_REQUESTS": {"OLLAMA_DEBUG_LOG_REQUESTS", DebugLogRequests(), "Log inference request bodies and replay curl commands to a temp directory"}, "OLLAMA_GO_TEMPLATE": {"OLLAMA_GO_TEMPLATE", GoTemplate(true), "Enable Modelfile TEMPLATE based rendering when available"}, "OLLAMA_FLASH_ATTENTION": {"OLLAMA_FLASH_ATTENTION", FlashAttention(false), "Enabled flash attention"}, "OLLAMA_KV_CACHE_TYP [source]
var ( // FlashAttention enables the experimental flash attention feature. FlashAttention = BoolWithDefault("OLLAMA_FLASH_ATTENTION") // GoTemplate enables Modelfile TEMPLATE rendering when a model has one. GoTemplate = BoolWithDefault("OLLAMA_GO_TEMPLATE") // DebugLogRequests logs inference requests to disk for replay/debugging. DebugLogRequests = Bool("OLLAMA_DEBUG_LOG_REQUESTS") // KvCacheType is the quantization type for the K/V cache. KvCacheType = String("OLLAMA_KV_CACHE_TYPE") // NoHistory disables readline history. NoHistory = Bool("OLLAMA_NOHISTORY") // NoPrune disables pruning of mode [source]
Vulkan is the documented path for GPUs that ROCm does not cover. Per the current docs it is enabled by default when the backend is installed. OLLAMA_VULKAN=0 or GGML_VK_VISIBLE_DEVICES=-1 disables all Vulkan GPUs. GGML_VK_VISIBLE_DEVICES takes numeric IDs to select specific Vulkan GPUs. [source]
On a mixed iGPU/dGPU system where the Vulkan iGPU is unstable, keep Vulkan on and set GGML_VK_VISIBLE_DEVICES to the discrete GPU index (for example 1 when Vulkan1 is the discrete GPU). [source]
When a new model does not fit, the scheduler picks the runner with shortest session duration (refCount 0 first), forces its sessionDuration to 0, unloads it and retries; queued requests are processed in order. If load OOMs with others resident, all other runners are evicted once before retry. [source]
A loaded runner is reloaded (not reused) if options, context shift, adapters, projector differ or the health ping fails, so varying num_ctx or other options per request between callers triggers reloads and cold starts. [source]
ROCR_VISIBLE_DEVICES selects AMD ROCm GPUs. List devices with `rocminfo`, and use the Uuid when available. An invalid ID such as -1 forces CPU. [source]
OLLAMA_MAX_QUEUE defaults to 512. The pending channel is buffered to that size; when full a request fails immediately with ErrMaxQueue ('server busy, please try again. maximum pending requests exceeded'), surfaced as HTTP 503. [source]
* `OLLAMA_MAX_LOADED_MODELS` - The maximum number of models that can be loaded concurrently provided they fit in available memory. The default is 3 \* the number of GPUs or 3 for CPU inference. * `OLLAMA_NUM_PARALLEL` - The maximum number of parallel requests each model will process at the same time, default 1. Required RAM will scale by `OLLAMA_NUM_PARALLEL` \* `OLLAMA_CONTEXT_LENGTH`. * `OLLAMA_MAX_QUEUE` - The maximum number of requests Ollama will queue when busy before rejecting additional requests. The default is 512 [source]
Ollama KV cache quantization is one server setting, OLLAMA_KV_CACHE_TYPE, with three documented values: f16 (the default), q8_0 and q4_0. It only works when flash attention is on, and it applies to every model the server loads. You set it as an Environment= line on the ollama systemd unit. The Ollama FAQ states the memory effect: q8_0 uses approximately half the memory of f16, and q4_0 approximately a quarter. Everything below was first checked against that FAQ and the Ollama 0.34.2 release in September 2026, and re-checked against the FAQ, the Ollama source and the llama.cpp source on 13 Octo [source]
OLLAMA_FLASH_ATTENTION is automatic by default: Ollama uses Flash Attention when the selected backend and devices support it. Setting OLLAMA_FLASH_ATTENTION=1 forces it on and OLLAMA_FLASH_ATTENTION=0 disables it. [source]
Yes. Pass each one with a -e flag at run time, for example docker run -e OLLAMA_KEEP_ALIVE=-1 -e OLLAMA_KV_CACHE_TYPE=q8_0 -p 11434:11434 ollama/ollama. They behave identically to a native install because the container runs the same ollama serve process. [source]
Launching ollama with the OLLAMA_VULKAN=1 environment variable set will now enable Vulkan API support as an alternative to the likes of AMD ROCm and NVIDIA CUDA acceleration. This is great for open-source Vulkan drivers, older AMD graphics cards lacking ROCm support, or even any AMD setup with the RADV driver present but not having installed ROCm. As we've seen when testing Llama.cpp with Vulkan, in some cases using Vulkan can be faster than using the likes of ROCm. [source]
Alternatively, you can change the amount of time all models are loaded into memory by setting the `OLLAMA_KEEP_ALIVE` environment variable when starting the Ollama server. The `OLLAMA_KEEP_ALIVE` variable uses the same parameter types as the `keep_alive` parameter types mentioned above. Refer to the section explaining [how to configure the Ollama server](#how-do-i-configure-ollama-server) to correctly set the environment variable. [source]
Parallel Requests per Model: Each loaded model can process several requests in parallel, controlled by the OLLAMA_NUM_PARALLEL setting (default is 1 or 4, depending on memory). [source]
Flash Attention is a feature of most modern models that can significantly reduce memory usage as the context size grows. Ollama uses Flash Attention automatically when the selected backend and devices support it. To force Flash Attention on, set `OLLAMA_FLASH_ATTENTION=1` when starting the Ollama server. To disable it, set `OLLAMA_FLASH_ATTENTION=0`. [source]
Parallel Processing: Ollama supports concurrent processing of requests. If the system has enough available memory (RAM for CPU inference, VRAM for GPU inference), multiple models can be loaded at once, and each loaded model can handle several requests in parallel. This is controlled by the environment variable OLLAMA_NUM_PARALLEL, which sets the maximum number of parallel requests each model can process simultaneously. By default, this is set to 4 (or 1, depending on memory availability), but it can be adjusted. [source]
// LogValue may run from goroutines that already hold refMu (see the scheduler // debug logs), so the unload-mutable fields are read only under TryLock and // omitted when the lock is contended. func (runner *runnerRef) LogValue() slog.Value { if runner == nil { return slog.StringValue("nil") } modelID := runner.modelPath if modelID == "" { modelID = runner.modelKey } attrs := []slog.Attr{} if runner.refMu.TryLock() { if runner.model != nil { attrs = append(attrs, slog.String("name", runner.model.Name)) } if len(runner.gpus) > 0 { attrs = append(attrs, slog.Any("inference", slices.Clone(runner [source]
No. The runner refuses a quantized value cache when flash attention is off and logs quantized V cache was requested, but this requires Flash Attention. Ollama 0.34 sets flash attention to auto and lets llama.cpp probe the backend, so set OLLAMA_FLASH_ATTENTION=1 in the same drop-in as the cache type and you remove the guess. If it still fails with it forced on, your backend has no flash attention kernel and the cache stays f16. [source]
When Ollama runs as the macOS application, set environment variables with `launchctl setenv` (e.g. `launchctl setenv OLLAMA_HOST "0.0.0.0:11434"`) and restart the Ollama app. The menu-bar app is launched by launchd, not the shell, so exports in ~/.zshrc are not seen. [source]
“I run out of VRAM when two requests arrive” [source]
On Windows, change the model directory by setting OLLAMA_MODELS as a user-account environment variable (Settings or Control Panel, 'Edit environment variables for your account'), then quit the tray app and relaunch it from the Start menu or open a new terminal. Same mechanism applies to OLLAMA_HOST. [source]
Batching in Ollama refers to the practice of grouping multiple incoming requests together and processing them as a unit. This allows for more efficient use of computational resources, especially when running on hardware that benefits from parallelized operations (such as GPUs). [source]
When multiple requests for the same model arrive simultaneously, Ollama can process them together in a batch if memory allows. This increases throughput and can reduce latency for each request, as the model can leverage optimized matrix operations over the batch. [source]
Ollama leverages both batching and parallel execution to process multiple requests efficiently. Batching groups requests for simultaneous processing, while parallel execution allows multiple requests (or models) to run concurrently. Both methods depend on system memory and are configurable for optimal performance. [source]
If Ollama runs as a systemd service, add Environment="OLLAMA_CONTEXT_LENGTH=4096" to the unit and restart it, or cap a single model from inside the session with /set parameter num_ctx 4096. Setting the variable in front of ollama run does nothing, because ollama run is only the client and the server holds the context. With the cache capped to fit, the same model on the same card held at 37 tokens per second. Same hardware, same weights, ten times the speed, decided entirely by whether the cache fit. [source]
Every number above assumes one request at a time. Serving multiple users changes the math, because the weights are shared but the KV cache is per request. Ten concurrent sessions do not load the model ten times, but they do allocate ten KV caches, and at long context that cache is the larger cost. The concurrent-requests selector in the calculator multiplies the cache accordingly. Budget VRAM for peak concurrency, not for a single chat. [source]
The worst case memory is the product of the first two. Two loaded models with four slots each is eight slot allocations of KV cache, all resident at once, and Ollama will try to satisfy that. On a single GPU box it is usually better to hold one model and give it slots, because the arithmetic stays something you can do in your head. [source]
When Ollama loads a model it launches a separate runner process. Two of the arguments it passes matter here: -c is the total context the runner allocates a KV cache for, and -np is the number of parallel sequences. Ollama sets -c to your per-request context length multiplied by the slot count. The runner then divides that total evenly between the slots, so each request still gets the context length you asked for. [source]
That is the whole constraint, and it is why parallelism is not free. Going from one slot to four asks for four times the KV cache at the same per-request context. Nothing is shared between slots, and an idle slot's share is not lent to a busy one, because the split is fixed when the runner starts. On a GPU that memory is VRAM. On a CPU only server it is system RAM, and the same multiplication applies. [source]
Yes. The server reads them at startup, and a running model keeps the slot count that was fixed into its runner process when it launched. Edit the drop-in with sudo systemctl edit ollama.service, then run sudo systemctl daemon-reload and sudo systemctl restart ollama. Confirm with systemctl show ollama --property=Environment, and then check the server config line in journalctl -u ollama, which lists the environment the server actually loaded. [source]
On mixed iGPU/dGPU systems where the Vulkan iGPU is unstable, keep Vulkan enabled and set `GGML_VK_VISIBLE_DEVICES` to the discrete GPU index. For example, use `GGML_VK_VISIBLE_DEVICES=1` when `Vulkan1` is the discrete GPU. [source]
Vulkan is bundled into the `ollama/ollama` image and is enabled by default when the container can access the GPU devices. [source]
If there is insufficient available memory to load a new model request while one or more models are already loaded, all new requests will be queued until the new model can be loaded. As prior models become idle, one or more will be unloaded to make room for the new model. Queued requests will be processed in order. When using GPU inference new models must be able to completely fit in VRAM to allow concurrent model loads. [source]
When loading a new model, Ollama evaluates the required VRAM for the model against what is currently available. If the model will entirely fit on any single GPU, Ollama will load the model on that GPU. This typically provides the best performance as it reduces the amount of data transferring across the PCI bus during inference. If the model does not fit entirely on one GPU, then it will be spread across all the available GPUs. [source]
The K/V context cache can be quantized to significantly reduce memory usage when Flash Attention is enabled. [source]
If raising OLLAMA_NUM_PARALLEL no longer keeps latency stable and the queue keeps growing under real traffic, that is one of the clearer signals worth weighing against a move to a purpose-built serving engine. Ollama to vLLM: When to Migrate Your Local LLM Server walks through that decision, including continuous batching and PagedAttention as the mechanisms vLLM uses to keep concurrent requests from degrading each other. [source]
if runnerToExpire == nil { // While we were performing load calculations, the loaded runner(s) unloaded in parallel // so findRunnerToUnload returned no runners. We'll try again and the loadedCount should be zero slog.Debug("runner to expire was nil, retrying") continue } // Trigger an expiration to unload once it's done runnerToExpire.refMu.Lock() slog.Debug("resetting model to expire immediately to make room", "runner", runnerToExpire, "refCount", runnerToExpire.refCount) if runnerToExpire.expireTimer != nil { runnerToExpire.expireTimer.Stop() runnerToExpire.expireTimer = nil } runnerToExpi [source]
Served in parallel. Every request reports a similar ttfb, and total rises for all of them together. The GPU is shared between the running slots, so each answer is slower than it would be alone while more of them finish per minute. This is the regime you are buying when you raise OLLAMA_NUM_PARALLEL. [source]
Match the method to how Ollama runs. Terminal server: your shell profile. macOS menu-bar app: launchctl, because launchd never reads ~/.zshrc. Linux: the systemd unit. Docker: a -e flag. Setting it in the wrong place is the number-one reason a variable looks ignored. [source]
That is 2.13 GiB against 4.0 GiB before, the 53 percent from the block arithmetic. The saving is what lets you raise num_ctx again, or fit a larger weight file, on the same box. The log line is the allocation. ollama ps is an estimate of the whole model and is the number the scheduler uses to decide whether a model fits at all, so watch both, and trust the log line when they disagree. [source]
Parallel request processing for a given model results in increasing the context size by the number of parallel requests. For example, a 2K context with 4 parallel requests will result in an 8K context and additional memory allocation. [source]
The light version is a reverse proxy. In nginx, limit_conn caps simultaneous connections and limit_req caps the arrival rate per client, so the overflow is refused at the proxy and never reaches Ollama. The heavy version is a job queue with a database in front of a worker that calls Ollama, which is what you want once requests must outlive a process restart. Sizing that for real traffic is its own exercise: planning a self hosted LLM for concurrent users goes through the arithmetic, and running Ollama on a VPS covers the base install these variables assume. [source]
Ollama's default context length depends on detected VRAM: under 24 GiB gives 4k, 24-48 GiB gives 32k, 48 GiB or more gives 256k. Larger context raises memory use; OLLAMA_CONTEXT_LENGTH on the server process or the app slider overrides it. [source]
// Wait for every unload event. Each runner produces exactly one // unloadedCh signal when its cleanup finishes. for range runnersToExpire { select { case <-ctx.Done(): slog.Debug("shutting down scheduler during evict-all wait") return false case <-s.unloadedCh: } } return true } [source]
A model that is already loaded keeps the slot count it was started with, because the value is fixed into the runner process at launch. The restart above unloads everything, so the next request reloads the model with the new setting and pays the load time once. How long a model stays resident after that is a separate control, covered in keeping an Ollama model loaded between requests. [source]
At a num_ctx of 32768 the f16 cache is 4.0 GiB. Doubling to 65536 doubles it to 8.0 GiB, and the model's full 131072 context needs 16.0 GiB for the cache before a single weight is loaded. The q8_0 and q4_0 columns sit a little above half and a quarter because of how the quantized blocks are laid out: every 32 values share one 16-bit scale, so a q8_0 block is 34 bytes and a q4_0 block is 18 bytes, against 64 bytes for f16. That is 53 percent and 28 percent, which is what the FAQ rounds to one half and one quarter. [source]
The Ollama installer creates a systemd unit, so the setting goes in a drop-in for that unit. The FAQ's instruction is exactly this. Set flash attention explicitly at the same time, because a quantized cache without it is refused by the runner, and auto leaves the decision to a probe you did not see. [source]
OLLAMA_KV_CACHE_TYPE:q8_0 means the server process has the variable. An empty result means it does not, whatever your shell says. [source]
When --flash-attn on is used with quantized KV cache types (e.g. --cache-type-k q8_0 --cache-type-v q4_0), and llama.cpp was built without -DGGML_CUDA_FA_ALL_QUANTS=ON, the server silently falls back to CPU-based attention. This results in: [source]
Ollama environment variables are settings the Ollama server reads at startup to control networking, model storage, memory behavior, and inference performance. Ollama v0.32.5 reads 26 OLLAMA_ variables plus the GPU selection and proxy variables it inherits from the environment. The server reads them once when it launches, so changing one only takes effect after you restart Ollama. [source]
When choosing hardware for Ollama, prioritize a single GPU with enough VRAM for the model whenever possible. Use a multi-GPU setup when you need to run larger models that exceed the memory capacity of one GPU. [source]
When running in a container, in some Linux distributions and container runtimes, the ollama process may be unable to access the GPU. Use `ls -lnd /dev/kfd /dev/dri /dev/dri/*` on the host system to determine the **numeric** group IDs on your system, and pass additional `--group-add ...` arguments to the container so it can access the required devices. For example, in the following output `crw-rw---- 1 0 44 226, 0 Sep 16 16:55 /dev/dri/card0` the group ID column is `44` [source]
When using the API, specify the `num_ctx` parameter: [source]
With the context left at its auto-sized default, generation dropped to roughly 3.6 tokens per second, a tenth of the real rate of the card. The fix is to cap the context so the cache stays inside VRAM. Ollama reads the context length from the server process, so set it there and restart: [source]
Administrators should monitor memory usage and adjust these parameters based on their hardware capabilities. Insufficient memory handling becomes crucial when running larger models (7B+ parameters) or processing multiple concurrent requests. [source]
Revert OLLAMA_HOST to 127.0.0.1:11434 in the drop-in when the proxy runs on the same host. Then create the credentials file: [source]
totalSize, vramSize := llama.MemorySize() trainContext := modelTrainContext(f) if effectiveNumCtx := llama.ContextLength(); req.model.ModelPath != "" && effectiveNumCtx > 0 { req.opts.NumCtx = effectiveNumCtx req.contextShift = resolveContextShift(req.shift, req.model) } runnerName := req.model.Runner if req.model.IsMLX() && runnerName == "" { runnerName = manifest.RunnerMLX } else if llm.IsLlamaCPP(llama) { // The loaded process is authoritative: a llama.cpp server may serve // the model even when the manifest metadata says ggml. runnerName = manifest.RunnerLlamaCPP } runner := &runnerRef{ mo [source]
// Only back off mmap when we still expect the model to fit on discrete GPU. // If VRAM is already tight, disabling mmap can make partial CPU offload // worse by turning file-backed mappings into anonymous memory. if predictedVRAM == 0 || availableVRAM == 0 || predictedVRAM > availableVRAM*80/100 { return false } [source]
// The refMu must already be held when calling unload func (runner *runnerRef) unload() { if runner.expireTimer != nil { runner.expireTimer.Stop() runner.expireTimer = nil } if runner.llama != nil { runner.llama.Close() } runner.model = nil runner.Options = nil runner.gpus = nil runner.contextShift = false } [source]
On Linux, Ollama runs as a systemd service. Running export OLLAMA_NUM_PARALLEL=4 in your shell changes nothing, because systemd starts the service with its own environment and never sees your shell. Use a drop-in file. [source]
systemctl show prints what systemd will hand the process. If your variable is missing there, the drop-in was not saved or daemon-reload was skipped. Confirm from the server's own side too: [source]
That message means the scheduling queue was full at the moment the request arrived. It says nothing about VRAM and nothing about the model. In current releases that queue only holds requests whose model has not been loaded yet. If the model is already resident when you run the loop above, expect every request to return 200 however low you set OLLAMA_MAX_QUEUE. The overflow shows up as a longer ttfb instead. [source]
Three exceptions to know before you trust it. OLLAMA_AUTH is declared in the source but has no entry in the map that backs the dump, so it never appears at all. OLLAMA_FLASH_ATTENTION prints false when unset even though the real default is automatic, as described above. OLLAMA_IGPU_ENABLE prints an empty string when unset, because it has no fixed default and the decision is made per device. For those three, the log tells you what you set, not what the server decided. [source]
Almost always because the variable is set in a different environment than the one the server runs in. The macOS app is started by launchd and never reads your shell profile, so it needs launchctl setenv. A systemd service ignores your shell and needs an Environment= line in the unit. Every method requires a full server restart, since Ollama reads the variables only at startup. If all of that checks out, confirm the variable still exists: OLLAMA_MAX_VRAM and OLLAMA_TMPDIR are both widely recommended and neither is read by current Ollama. [source]
Model memory adds up too when concurrent requests use different models. [source]
On Linux, the documented install is `curl -fsSL https://ollama.com/install.sh | sh`; the manual path extracts ollama-linux-amd64.tar.zst (plus ollama-linux-amd64-rocm.tar.zst for AMD, or the arm64 package) into /usr. When upgrading manually, remove /usr/lib/ollama first. [source]
This comes up when a model almost fits in VRAM and the cache is what pushes it over. The control people are told to reach for, --no-kv-offload, is a llama.cpp flag, and Ollama never passes it: the argument list Ollama builds for llama-server includes the adjacent --no-mmproj-offload but not this one. Two attempts to surface it went nowhere. PR #9751 has been a draft since 2025-03-26, and an earlier attempt, PR #7274, was closed unmerged on 2025-08-08. The tracking issue, ollama/ollama#9750, is still open but has had no activity since the day it was filed in March 2025. [source]
Mixture-of-experts models read more per token than their active parameter count suggests, because attention, shared experts and the router are stored at higher precision than the routed experts. Across Gemma 4 26B-A4B, Qwen3-Coder 30B-A3B and gpt-oss-20b on an RTX 4090, the measured reads came out 1.63 times the naive file size × active ÷ total. DeepSeek V4 Flash reads 2.8 times, calibrated to a published DGX Spark run at 19.1 tok/s. The calculator does not model multi-token prediction drafting, which Ollama turns on by default for Qwen3.8, Qwen3.6 and Gemma 4 and which roughly doubled their s [source]
A line showing 127.0.0.1:11434 means the drop-in was not picked up. Check what systemd actually passes to the process: [source]
// Default automatic value for number of models we allow per GPU // Model will still need to fit in VRAM, but loading many small models // on a large GPU can cause stalling var defaultModelsPerGPU = 3 [source]
The distinction that trips people up: these configure the background server (ollama serve), not the ollama run command. Setting one in the same shell where you type ollama run does nothing if the server is already running as a separate process or a macOS app. The server has its own environment, and that is the one that counts. [source]
That last point is worth stating plainly because most "what's new in Ollama" posts imply otherwise. Between mid-June and late July 2026, zero environment variables changed. What did change was runtime behavior and documentation, and one of those changes invalidates advice that is still repeated almost everywhere, including in the earlier version of this page. It is covered in the flash attention section below. [source]
If you experience gibberish responses when models load across multiple AMD GPUs on Linux, see the following guide. [source]
The longer your context window, the more VRAM Ollama needs because it keeps more of your prompt and conversation in memory while generating a response. [source]
When quantisation is applied to the K/V context cache it greatly reduces the memory requirements, allowing users to run larger models or use larger context sizes on their existing hardware. [source]
Significantly reduced crashes due to out of memory issues: Because memory management is exact, over-allocations no longer occur meaning fewer out of memory issues. [source]

How-to and procedures

To resolve this, upgrade to the ROCm v7 driver using the `amdgpu-install` utility from [AMD's ROCm documentation](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/). After upgrading, reboot and restart Ollama. [source]
To select specific Vulkan GPU(s), you can set the environment variable `GGML_VK_VISIBLE_DEVICES` to one or more numeric IDs on the Ollama server as described in the [FAQ](faq#how-do-i-configure-ollama-server). If you encounter any problems with Vulkan based GPUs, you can disable all Vulkan GPUs by setting `OLLAMA_VULKAN=0` or `GGML_VK_VISIBLE_DEVICES=-1`. [source]
Use `OLLAMA_VULKAN=0` to disable Vulkan, or `GGML_VK_VISIBLE_DEVICES=<ids>` to select specific Vulkan devices. [source]
To run Ollama using Docker with AMD GPUs, use the `rocm` tag and the following command: [source]
Set OLLAMA_KV_CACHE_TYPE=q8_0 to roughly halve the memory the context cache uses. It takes effect only when flash attention is active, which on supported hardware it now is by default. [source]
To set server environment variables under systemd, run `sudo systemctl edit ollama.service` (creates /etc/systemd/system/ollama.service.d/override.conf), add `[Service]` plus `Environment="NAME=value"` lines, then `systemctl daemon-reload` and `systemctl restart ollama`. Do not edit the packaged unit file; package updates overwrite it. [source]
Install Ollama on Ubuntu 24.04, run it as a systemd service, configure OLLAMA_HOST and OLLAMA_MODELS, and verify the API on port 11434. [source]
Set OLLAMA_KV_CACHE_TYPE with systemctl edit [source]
Set Ollama environment variables in your shell profile for terminal use, with launchctl setenv for the macOS app, in the systemd unit on Linux, or with -e flags for Docker. Ollama reads them at server startup, so restart Ollama after any change. [source]
Set OLLAMA_KEEP_ALIVE to control how long a model stays in memory after its last request. The default is 5 minutes. Any negative number keeps models loaded indefinitely, and 0 unloads immediately after each request. [source]
Use the `ollama ps` command to see what models are currently loaded into memory. [source]
Create a service file in `/etc/systemd/system/ollama.service`: [source]
To submit images to multimodal models such as llava or bakllava, provide a list of base64-encoded images: [source]
Use a method that survives reboots. launchctl setenv on macOS applies to the current login session, so for a permanent macOS setting add it to a login script or LaunchAgent. On Linux, the systemd unit override is already permanent. In a shell, an export line in ~/.zshrc or ~/.bashrc persists across sessions. [source]
Run Open WebUI as a browser front end for Ollama on Ubuntu 24.04: Docker setup, secure API binding, model pulls and an HTTPS reverse proxy. [source]
Check the effect in ollama ps and the runner log [source]
Run ollama ps to confirm 100% GPU, then repeat the three-run measurement. [source]
Run that with one slot, then again at the concurrency you actually expect, and compare the two numbers that decide whether users are happy: time to first token, and tokens per second per request. Throughput per request always falls as slots are added. The question is whether it falls further than your users will accept. Measuring tokens per second on a local LLM covers the method in more detail, including how to keep the prompt constant between runs. [source]
Add -DGGML_CUDA_FA_ALL_QUANTS=ON to the default CMake configuration. The build time increase is a one-time cost that most users would accept for 25x performance with quantized KV cache. [source]
Check Ollama’s hardware support page first to see whether it supports your exact GPU, then use its VRAM or unified memory to determine which models you can run entirely on it. [source]
To run Ollama on Hostinger GPU hosting, choose a GPU with enough VRAM for the model you plan to use, deploy Ollama in Hostinger dashboard, pull and run your model, and verify that Ollama is using the GPU. [source]
Run the same benchmark with num_gpu changed to -1: [source]
Use `OLLAMA_VERSION` environment variable with the install script to install a specific version of Ollama, including pre-releases. You can find the version numbers in the [releases page](https://github.com/ollama/ollama/releases). [source]

Examples and snippets

``` — `Thinking: Okay, the user is asking about Ollama's new engine. I need to figure out what they're referring to.` … [source]
```shell theme={"system"} — `sudo nvidia-ctk runtime configure --runtime=docker` … [source]
```shell theme={"system"} — `docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/` … [source]
```shell theme={"system"} — `OLLAMA_CONTEXT_LENGTH=8192 ollama serve` … [source]
``` — `OLLAMA_CONTEXT_LENGTH=64000 ollama serve` … [source]
``` — `# Allow all Chrome, Firefox, and Safari extensions` … [source]
``` — `NAME ID SIZE PROCESSOR CONTEXT UNTIL` … [source]
```powershell theme={"system"} — `$env:OLLAMA_DEBUG="1"` … [source]

Measurements and reference values

Memory, specifically, scales with OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH: four parallel slots at a 32K context setting reserve KV cache as if a single 128K sequence were loaded, even before any request uses it. On a 16 GB card that budget math usually matters more than the queueing behavior — see KV Cache on 16 GB GPUs for the full VRAM budget and why OLLAMA_NUM_PARALLEL=1 is usually the right starting point for a single long-context session. [source]
Ollama exposes no queue-depth metric; ollama ps and /api/ps list loaded models only, so queueing must be inferred client-side from time-to-first-byte or counted 503s. [source]
Diagnose offload with `ollama ps`: the PROCESSOR column shows 100% GPU or a CPU/GPU split; with the new scheduler the vendor says nvidia-smi figures should match ollama ps. [source]
Three things consume VRAM during inference. The weights are the big, fixed cost. The KV cache grows with context length and with the number of concurrent requests. Overhead is a smaller tax for the CUDA or Metal context and compute buffers, about 0.7 GB at short context and roughly 1.5 GB at 256K, paid once per GPU. [source]
Independent single-16GB-GPU test on v0.12.1: for models too big for VRAM, three of four (mistral-small3.2:24b, gemma3:27b, qwen3:32b) shifted slightly more to CPU (e.g. 41%->44%, 28%->29%, 30%->32% CPU); qwen3:30b-a3b unchanged. Contradicts the vendor claim; anecdotal, one machine, unnamed old version. [source]
go func() { // typical convergence is 0.5-1.5s - If it takes too long to discover and converge, let the scheduler estimate VRAM usage ctx, cancel := context.WithTimeout(context.Background(), s.waitForRecovery) defer cancel() ticker := time.NewTicker(250 * time.Millisecond) defer ticker.Stop() for { select { case <-ticker.C: // Query GPUs, look for free to go back up gpusNow := s.getGpuFn(ctx, runners) totalMemoryNow = 0 freeMemoryNow = 0 for _, gpu := range gpusNow { totalMemoryNow += gpu.TotalMemory freeMemoryNow += gpu.FreeMemory } if freeMemoryNow > freeMemoryBefore { logutil.Trace("gpu VRA [source]
LLM performance comparison - Ollama on 16GB VRAM GPU [source]
| Compute Capability | Family | Cards | | - | - | - | | 12.1 | NVIDIA | `GB10 (DGX Spark)` | | 12.0 | GeForce RTX 50xx | `RTX 5060` `RTX 5060 Ti` `RTX 5070` `RTX 5070 Ti` `RTX 5080` `RTX 5090` | | | NVIDIA Professional | `RTX PRO 4000 Blackwell` `RTX PRO 4500 Blackwell` `RTX PRO 5000 Blackwell` `RTX PRO 6000 Blackwell` | | 9.0 | NVIDIA | `H200` `H100` | | 8.9 | GeForce RTX 40xx | `RTX 4090` `RTX 4080 SUPER` `RTX 4080` `RTX 4070 Ti SUPER` `RTX 4070 Ti` `RTX 4070 SUPER` `RTX 4070` `RTX 4060 Ti` `RTX 4060` | | | NVIDIA Professional | `L4` `L40` `RTX 6000` | | 8.6 | GeForce RTX 30xx | `RTX 3090 Ti [source]
Ollama GPU requirements range from about 3‑4 GB of VRAM for a small 3B model to around 50 GB for a 70B model, at Q4_K_M quantization (a widely used compressed version of a model) with an 8K context. [source]
The Q4_K_M version fits on an 8 GB GPU with VRAM to spare for context. The Q8_0 version is already 8.5 GB before adding context, so the same GPU can’t hold the entire workload in VRAM. [source]
Behind an nginx reverse proxy, rewrite the Host header to localhost:11434 (or list the external hostname in OLLAMA_ORIGINS) to avoid 403 Forbidden; turn off proxy_buffering and raise proxy_read_timeout so streamed generations are not cut at ~60 s. [source]
A concrete case from testing: the 32B model on a 24 GB RTX 3090. The weights are about 20 GB and fit. But recent Ollama versions size the context window to the VRAM they detect, and a 24 GB card is handed a 32K window by default. That 32K KV cache on top of the weights pushes the total past 24 GB, the overflow spills to the CPU, and throughput collapses: [source]
The number of concurrently loaded models is limited by OLLAMA_MAX_LOADED_MODELS (default: 3×GPU count or 3 for CPU). [source]
OLLAMA_MAX_LOADED_MODELS Maximum concurrently loaded models 3×GPU count or 3 [source]
GPU: NVIDIA RTX 5070 Ti Laptop (Blackwell sm_120, 12 GB VRAM) [source]
===================================== ROCm System Management Interface ===================================== =============================================== Concise Info =============================================== Device [Model : Revision] Temp Power Partitions SCLK MCLK Fan Perf PwrCap VRAM% GPU% Name (20 chars) (Edge) (Avg) (Mem, Compute) ============================================================================================================ 0 [0x5026 : 0xc1] 40.0°C 5.0W N/A, N/A 700Mhz 96Mhz 0% auto 164.0W 10% 5% 0x73ef =============================================================== [source]
Q4_K_M file sizes: Llama 3.1 8B about 4.9 GB, Qwen3 32B about 19.8-20 GB, Llama 3.3 70B about 42.5-43 GB. Q8_0 of Llama 3.1 8B is about 8.5 GB, roughly double Q4_K_M, and Q6_K is about 6.6 GB. [source]
At Q4_K_M quantization and an 8K context, Ollama needs roughly 3-4 GB of VRAM for 1B-4B models, 6-7 GB for 7B-8B models, and 47-52 GB for 70B-class models. [source]
But when I ask the model something through vscode and check ollama ps, it tells me that it split the model 76%/24% CPU/GPU and that the model actually needs 8.9 GB, which is quite a lot more than the 2.5 gb advertized. When I drop the context window down to 4000, ollama reports memory size of 3.3 GB, which should fit in the 4GB of VRAM, but it still splits it 51%/49% CPU/GPU. [source]
Lemonade Fixes AMD APU Model Streaming, Drops OpenMOSS ROCm As ~40x Slower Than Vulkan [source]
Start at 1 and raise it one step at a time. After each step, restart Ollama, send one request to load the model, and run ollama ps. Stop at the last value where PROCESSOR still reads 100% GPU and the SIZE column leaves headroom for the longest context you serve. Then measure time to first token and tokens per second at that setting under your real concurrency, and back off one step if per request speed has dropped below what your users tolerate. [source]
time=2024-03-02T00:34:13.785Z level=INFO source=images.go:710 msg="total blobs: 16" time=2024-03-02T00:34:13.785Z level=INFO source=images.go:717 msg="total unused blobs removed: 0" time=2024-03-02T00:34:13.786Z level=INFO source=routes.go:1019 msg="Listening on [::]:11434 (version 0.1.27)" time=2024-03-02T00:34:13.786Z level=INFO source=payload_common.go:107 msg="Extracting dynamic libraries..." time=2024-03-02T00:34:16.164Z level=INFO source=payload_common.go:146 msg="Dynamic LLM libraries [rocm_v6 cuda_v11 cpu cpu_avx rocm_v5 cpu_avx2]" time=2024-03-02T00:34:16.164Z level=INFO source=gpu.go [source]
Four concurrent requests need four separate 2K contexts, so Ollama is holding roughly 8K tokens of context across them. That raises total VRAM usage to about 6-7 GB, even though each request is still set to 2K. [source]
Say you’re running Llama 3.1 8B Q4_K_M with a 2K context. One request uses about 5-6 GB of VRAM in total. [source]
Update: It keeps getting weirder. So I found a model that fits completely into memory, qwen3.5:0.8b. But now it just sits there. The VRAM gets filled but nvtop tells me that there is 0-5% gpu utilization while ollama ps tells me the model is completely run on gpu. [source]
Keep in mind that system RAM and VRAM are separate pools of memory. Your 16 GB of RAM still serves the operating system and other apps, while the GPU has its own 8 GB of VRAM for Ollama. [source]
Context and runtime memory sit on top of the model file, which is why a 4.9 GB model needs more than 4.9 GB of VRAM to run entirely on the GPU. The sections below break down each factor and how much VRAM it adds. [source]
For example, a model that needs about 48 GB of VRAM could run across two GPUs when neither GPU has enough memory to hold it independently. However, combining GPUs does not make their VRAM behave exactly like one larger memory pool, and transferring data between GPUs can affect performance. [source]
Qwen3.8 27B UD-Q4_K_XL, 256K context, q4_0 cache, RTX 409024.0 GB, fitsfits, 22.2 GiB (23.8 GB) in use [source]
GPU: NVIDIA RTX 3080/4090 with 16GB+ VRAM for larger models [source]
GPU: 1x NVIDIA GeForce RTX 4090 Model: gemma3:12b Context length: 128k [source]
server { listen 443 ssl; server_name ollama.example.com; ssl_certificate /etc/letsencrypt/live/ollama.example.com/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/ollama.example.com/privkey.pem; client_max_body_size 64m; location / { auth_basic "ollama"; auth_basic_user_file /etc/nginx/ollama.htpasswd; proxy_pass http://127.0.0.1:11434; proxy_set_header Host 127.0.0.1:11434; proxy_http_version 1.1; proxy_buffering off; proxy_read_timeout 600s; } } [source]
ollama ps prints one row per loaded model with the columns NAME, ID, SIZE, PROCESSOR, CONTEXT and UNTIL. Confirm CONTEXT reads 32768; if it shows a smaller number the option was not applied and the rest of the test is meaningless. Write down SIZE. That is Ollama's estimate of everything the model occupies, weights and cache together. On a VPS without a GPU, PROCESSOR reads 100% CPU, which is expected and is not a problem. [source]
CPU-only inference holds the model in system RAM. Guidance: about 16 GB RAM for a 4.9 GB 8B Q4 model, 32 GB as headroom for a ~20 GB model paired with an 8 GB GPU. Apple Silicon uses unified memory; about 23-25 GB is usable by Ollama on a 32 GB Mac. [source]
Tier guide at Q4_K_M, 8K context, one request: 7-8B needs about 6-7 GB VRAM (8 GB card), 12-14B about 10-12 GB (16 GB card), 20-32B about 19-24 GB (24 GB card with short context), 70-72B about 47-52 GB (48 GB tight, 80 GB comfortable). [source]
The two lines worth memorizing: 24 GB is the least you want for a 27B to 32B model, and 48 GB is the entry ticket to a 70B on a single card. The 32 GB RTX 5090 is the fastest single card here, but it is still one tier short of a 70B at Q4_K_M, which lands at 42.5 GB before context. At about $7,200 against about $8,000 for the 48 GB RTX 6000 Ada, the step up to 70B now costs only about a tenth more than the fastest 32 GB card, though the 6000 Ada generates tokens at a little over half the speed of the 5090. [source]
Some 32B Q4_K_M models fit on a 24 GB GPU, but leave little room as the context grows. A 48 GB GPU is similarly tight for many 70B models, so 80-96 GB gives you much more room. [source]
• Q8_0 K/V: Around 3GB, (50%~ saving). [source]
• Q4_0 K/V: Around 2GB, (66%~ saving). [source]
Nemotron 3.5 Lightning 30B-A3B18.9 GB (Q4_0)6 KiB + 0.05 GB state0.3 GB0.9 GB19.9 GB [source]
32B dense32 GBRTX 5090, or 24 GB with a q4_0 cache [source]
This is the variable people reach for after the first time a model unloads mid-session and the next prompt takes ten seconds to reload from disk. A negative value trades RAM for latency: the model sits in memory until you stop the server. On a machine you use for AI all day, that trade is usually worth it. On a 16GB Mac where you also need memory for everything else, the 5-minute default exists for a reason. [source]
That means the same model that fits on an 8 GB GPU with a short context no longer fits entirely in VRAM at 32K. Longer contexts are useful when you want Ollama to keep track of more information, such as a long conversation or a large document. [source]
Match the card to the largest model you intend to run at Q4_K_M, then leave a tier of headroom for context. The prices below were read from Amazon US on 10 October 2026 and sit far above list because of the ongoing memory shortage, so treat them as a band and check the live price. A 5090 at about $7,200 is more than three times its $1,999 launch price, and a new-stock 3090 at about $1,900 is still the cheapest way into the 24 GB tier. [source]
48 GBRTX 6000 Adaabout $8,00070B at Q4_K_M with a 16K context [source]
Take Llama 3.1 8B Q4_K_M as an example. Its model file is about 4.9 GB. [source]

Problems, failure modes and limitations

Vulkan cannot read free VRAM unless Ollama runs as root or the binary has cap_perfmon (`sudo setcap cap_perfmon+ep /usr/local/bin/ollama`). Without either, the scheduler uses approximate model sizes. On some Linux distributions the ollama user must also join the render group for AMD GPUs. [source]
CUDA_VISIBLE_DEVICES takes a comma-separated list to limit Ollama to a subset of NVIDIA GPUs. UUIDs (from `nvidia-smi -L`) are more reliable than numeric IDs because ordering may vary. An invalid ID such as -1 forces CPU. [source]
// Now that we've summed up all the GPU usage predictions across all the loaded runners, update the gpu list for i := range allGpus { if p, ok := predMap[allGpus[i].DeviceID]; ok { slog.Debug("gpu reported", "gpu", allGpus[i].ID, "library", allGpus[i].Library, "available", format.HumanBytes2(allGpus[i].FreeMemory)) if p > allGpus[i].TotalMemory { // Shouldn't happen slog.Warn("predicted usage exceeds VRAM", "gpu", allGpus[i].ID, "totalMemory", allGpus[i].TotalMemory, "predicted", p) allGpus[i].FreeMemory = 0 } else if (allGpus[i].TotalMemory - p) < allGpus[i].FreeMemory { // predicted free is [source]
Hardware support at introduction: Metal on Apple Silicon, CUDA on Pascal and newer, and most ROCm GPUs. If a model or hardware does not support flash attention, Ollama falls back to f16 and logs a warning. [source]
On Linux after a suspend/resume cycle, Ollama can fail to find the NVIDIA GPU and fall back to CPU. Reloading the UVM driver works around it: `sudo rmmod nvidia_uvm && sudo modprobe nvidia_uvm`. [source]
A quantized K/V cache requires Flash Attention. Without it the runner rejects a quantized V cache with the error 'quantized V cache was requested, but this requires Flash Attention'. Fix: set OLLAMA_FLASH_ATTENTION=1 explicitly; if the error persists the backend has no flash attention kernel and the cache must stay f16. [source]
OLLAMA_KV_CACHE_TYPE is a global option: every model runs with the same type. K and V use the same type, and the type cannot be set per model, in a Modelfile, via the API or on the command line. For mixed types the SSD Nodes guide suggests a second Ollama instance on another port. [source]
Quality measurements are early and come from the feature author's blog: on Qwen 2.5 Coder 7B (2024-12-06) perplexity went from 8.3891 (f16) to 8.3934 (q8_0), +0.0043. Early implementation figures were roughly +0.002 to +0.05 for q8_0 and +0.206 to +0.25 for q4_0. The author flags embedding models (Ollama tries to detect them and disables quantisation, with a warning), vision models and high-attention-head models (Qwen 2) as possibly more sensitive. [source]
Upstream llama.cpp CUDA builds compiled without GGML_CUDA_FA_ALL_QUANTS can reportedly fall back silently to CPU attention when flash attention is combined with some quantized K/V combinations (reported: q8_0 K with q4_0 V, head size 512), giving 25-45x slower prefill and no warning. The report is one user's setup, was closed 'not planned' on 2026-07-26 and does not name Ollama builds. [source]
At 4K context the KV cache is a rounding error against the weights. At 128K it can exceed them: Llama 3.1 8B needs 4.9 GB of weights and 17.2 GB of cache at that length. The cache is allocated up front for the maximum context you configure, and it grows linearly with that length and with every concurrent request. Three levers keep long context affordable. The model architecture decides the first two, grouped-query attention and hybrid layers that skip the cache altogether. The third is yours: KV cache quantization to 8-bit or 4-bit, which trades a little quality for a much smaller cache. If a [source]
Tip: If you also control your client, add retries with jitter and keep-alive connections. Many “Ollama is slow” issues are really queueing + connection overhead. [source]
When requests exceed the parallelism limit, they are queued (FIFO) up to OLLAMA_MAX_QUEUE. [source]
The model unloads between your two measurements. The default keep-alive is five minutes, so a slow second command can hit an empty ollama ps. Pass "keep_alive": "30m" in the request, or keep the model loaded for the whole session. [source]
* `AMD_LOG_LEVEL=3` Enable info log levels in the AMD HIP/ROCm libraries. This can help show more detailed error codes that can help troubleshoot problems * `OLLAMA_DEBUG=1` During GPU discovery additional information will be reported * Check dmesg for any errors from amdgpu or kfd drivers `sudo dmesg | grep -i amdgpu` and `sudo dmesg | grep -i kfd` [source]
The FAQ describes q8_0 as "a very small loss in precision" and q4_0 as "a small-medium loss in precision that may be more noticeable at higher context sizes". Take the second half of that sentence seriously. Weight quantization and KV cache quantization are both called quantization, but they are different operations with different failure modes. [source]
The default is safe, and it is also why a second user reports that the server "hung" when nothing is broken. Adding slots is a two line change. The part that bites is memory. Every parallel slot needs its own key/value cache (KV cache), the block of memory a model keeps for the tokens it has already processed. Add slots without adding VRAM (video memory on the GPU) and you turn a slow answer into a failed load. On a CPU only server the same cache comes out of system RAM instead. [source]
In llama-server, when --flash-attn on is set with a quantized KV cache type, check if the appropriate CUDA FA kernel was compiled. If not, emit a clear warning: [source]
Related: Misc. bug: Asymmetric K/V cache quantization types cannot be offloaded to GPU #20866, Eval bug: Mistral 4 119b - flash attention not working properly #20710 [source]
Set OLLAMA_HOST=0.0.0.0:11434 to bind Ollama to all network interfaces, and OLLAMA_ORIGINS to allow cross-origin browser requests. By default Ollama binds only to 127.0.0.1, so other machines cannot reach it. [source]
Warning: Flash attention with quantized KV cache requires building with -DGGML_CUDA_FA_ALL_QUANTS=ON. Using CPU fallback for attention which will be extremely slow. Consider using --cache-type-k f16 --cache-type-v f16 instead. [source]
When a model plus its KV cache exceed VRAM, the runtime does not fail. It keeps the overflow layers in system RAM and computes them on the CPU. That sounds graceful and is not. GPU memory moves at 500 to 1800 gigabytes per second; DDR5 system memory moves at 50 to 80, and an 8-core cloud VM running DeepSeek V4 Flash entirely on the CPU turned that into about 25 GB/s of effective throughput. Every offloaded layer drags token generation down to that rate, and the result is a cliff, not a slope. [source]
That line holds the full runner command line, including -c and -np. If -np is 1 after you set the variable, there are two possible causes. The first is the model. Ollama limits some model architectures to one slot no matter what you set, and logs model architecture does not currently support parallel requests when it does. As of October 2026 that list includes the Qwen 3.5, Qwen3-VL, Qwen3-Next, LFM2 and Nemotron-H families, and embedding models always get one slot. Run journalctl -u ollama --no-pager | grep "does not currently support parallel" to check. If that warning is absent, the setting [source]
Because OLLAMA_NUM_PARALLEL defaults to 1, so a loaded model runs one request at a time and the rest wait in order. The waiting request holds its HTTP connection open and sends no bytes until a slot frees, which looks identical to a slow model from the client side. The tell is the shape of the timing: a long pause and then text at full speed is a queue, while a slow trickle from the first token is a slow model. Raise the slot count with a systemd drop-in and restart the service. Then check the -np value on the runner line, because some model architectures are held to one slot no matter what yo [source]
If you have multiple NVIDIA GPUs in your system and want to limit Ollama to use a subset, you can set `CUDA_VISIBLE_DEVICES` to a comma separated list of GPUs. Numeric IDs may be used, however ordering may vary, so UUIDs are more reliable. You can discover the UUID of your GPUs by running `nvidia-smi -L` If you want to ignore the GPUs and force CPU usage, use an invalid GPU ID (e.g., "-1") [source]
When Ollama starts up, it takes inventory of the GPUs present in the system to determine compatibility and how much VRAM is available. Sometimes this discovery can fail to find your GPUs. In general, running the latest driver will yield the best results. [source]
A smaller queue turns that overflow into an immediate 503, but only for requests that are waiting for a model to load, such as the burst that arrives after a restart or while models are being swapped. As of October 2026, requests to a model that is already loaded bypass OLLAMA_MAX_QUEUE and wait for a free slot with no limit. Lowering the queue therefore will not shorten the wait on a busy server with one resident model. To get a fast refusal there, cap concurrency in front of Ollama, as the next section describes. A 503 is useful: a reverse proxy can retry it, a client can back off, a dashboa [source]
If too many requests are sent to the server, it will respond with a 503 error indicating the server is overloaded. You can adjust how many requests may be queued by setting `OLLAMA_MAX_QUEUE`. [source]
slog.Warn("runtime OOM detected; expiring loaded models to clear memory before next request", "model", schedulerModelKey(model), "error", err) for _, runner := range runners { runner.refMu.Lock() if runner.expireTimer != nil { runner.expireTimer.Stop() runner.expireTimer = nil } runner.sessionDuration = 0 if runner.refCount <= 0 { s.expiredCh <- runner } runner.refMu.Unlock() } } [source]
My PR integrated that functionality into Ollama which involved supporting the required configuration, implementing memory estimations for layer placement, error and condition handling, ensuring compatibility with the existing codebase, a lot of testing and a lot of communication around the feature to help others understand it's importance and how to use it. [source]
func (s *Scheduler) processCompleted(ctx context.Context) { // Process completed requests, expired timers, and unloading models for { select { case <-ctx.Done(): slog.Debug("shutting down scheduler completed loop") return case finished := <-s.finishedReqCh: finishedKey := schedulerModelKey(finished.model) s.loadedMu.Lock() runner := s.loaded[finishedKey] s.loadedMu.Unlock() if runner == nil { slog.Error("finished request signal received after model unloaded", "modelPath", finishedKey) continue } runner.refMu.Lock() runner.refCount-- if runner.refCount <= 0 { if runner.sessionDuration <= 0 { sl [source]
Warning: The unit file, source configuration file or drop-ins of ollama.service changed on disk. Run 'systemctl daemon-reload' to reload units. Printed by systemctl restart when you skipped daemon-reload. The restart used the old environment. Reload, then restart again. [source]
Slow local vision models can need a longer image-understanding timeout than cloud models, and can crash on constrained hardware if Ollama tries to allocate the model's full advertised vision context. Set a capability timeout and cap `num_ctx`: [source]
The two are separate problems. OLLAMA_HOST controls which network interface the server binds to. OLLAMA_ORIGINS controls which web origins the browser is allowed to call. If you are wiring a web app to Ollama and getting a silent CORS failure, the host is fine and the origins are the issue. We walk through that exact fix in the Ollama CORS fix on Mac. [source]
Start the server with OLLAMA_DEBUG=1 and read the startup log. If it shows your value, the setting is live. If it shows the default, you set it in an environment the server does not see, so go back to step 1. Three variables are exceptions and cannot be confirmed this way: OLLAMA_AUTH, OLLAMA_FLASH_ATTENTION, and OLLAMA_IGPU_ENABLE. [source]
Sometimes the Ollama can have difficulties initializing the GPU. When you check the server logs, this can show up as various error codes, such as "3" (not initialized), "46" (device unavailable), "100" (no device), "999" (unknown), or others. The following troubleshooting techniques may help resolve the problem [source]
On linux, AMD GPU access typically requires `video` and/or `render` group membership to access the `/dev/kfd` device. If permissions are not set up correctly, Ollama will detect this and report an error in the server log. [source]
Slow local vision models may need a longer client-side image timeout than cloud models; the client timeout is separate from the Ollama HTTP request guard. [source]
// OOM retry path: load() crashed post-spawn and we still // have other models resident. Evict all of them, wait for // every unload, then loop back to retry the load once. // load() has already set oomRetryAttempted so a second // crash falls through to the fail-fast path. if pending.oomRetryAttempted { if !s.evictAllAndWait(ctx, pendingKey) { return } continue } [source]
Apple silicon, the AMD Strix Halo and the NVIDIA GB10 share one memory pool between CPU and GPU, and the GPU uses it directly as VRAM. That changes the capacity math. A 128 GB machine holds DeepSeek V4 Flash at its 90.9 GB UD-IQ2_M tier, which no consumer discrete GPU can fit, and the mini PCs built to hold one are compared separately. Two caveats keep it honest. The GPU cannot map the whole pool. macOS allows about two thirds of RAM on smaller Macs and three quarters on larger ones, so a 64 GB Mac behaves like a 48 GB GPU until the limit is raised with sudo sysctl iogpu.wired_limit_mb. On Str [source]
func (s *Scheduler) getRunner(c context.Context, m *Model, opts api.Options, sessionDuration *api.Duration, numCtxAuto bool, numBatchAuto bool, shift *bool) (chan *runnerRef, chan error) { if opts.NumCtx < 4 { opts.NumCtx = 4 } [source]
OLLAMA_MAX_QUEUE is how many requests may sit waiting to be scheduled, which in practice means waiting for a model to be loaded. The default is 512. The request that arrives when that queue is full is rejected straight away. Requests for a model that is already loaded skip this queue and wait for a free slot instead, and that wait has no limit. Ollama's maintainers track this as a feature request (issue #7758), which was still open in October 2026. [source]
There is a limit you cannot tune your way past. Ollama splits the KV cache into equal, fixed slots when the model loads. An idle slot's memory cannot be used by a busy one, and the slot count cannot change without unloading the model. That design is a good fit for one person, or a small team, or a coding agent. [source]
Setting OLLAMA_HOST=0.0.0.0:11434 puts the API on every interface, and Ollama has no built-in authentication. With the default queue, an open endpoint will accept up to 512 requests waiting for a model to load, from anyone who finds it. For a model that is already loaded, it will hold every request that connects, with no limit. Tying the server up this way costs an attacker almost nothing: long prompts, no login, no rate limit, no bill. Your own users then get 503 responses or long waits, and the machine is busy the whole time. [source]
rocBLAS error: Cannot read /opt/rocm/lib/rocblas/library/TensileLibrary.dat: Illegal seek for GPU arch : gfx1032 List of available TensileLibrary Files : "/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1101.dat" "/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1030.dat" "/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx90a.dat" "/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx942.dat" "/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx941.dat" "/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx1102.dat" "/opt/rocm/lib/rocblas/library/TensileLibrary_lazy_gfx940.dat" "/ [source]
If you are experiencing problems getting Ollama to correctly discover or use your GPU for inference, the following may help isolate the failure. [source]
As stated in the FAQ, it is not recommended to use K/V cache quantisation for embedding models as these are more sensitive to quantisation. For this reason Ollama attempts to detect embedding models and disable K/V cache quantisation automatically (you'll see a warning in the logs if this happens). [source]
While practically all modern models support Flash Attention, if a model is loaded that - or if your hardware doesn't support Flash Attention, Ollama will automatically fall back to the default F16 quantisation. You'll see a warning in the logs if this happens. [source]
Queue Limits: The maximum number of queued requests is controlled by OLLAMA_MAX_QUEUE (default: 512). If the queue is full, new requests receive a 503 error indicating the server is overloaded. [source]
The firewall allows the port, but connections still fail. OLLAMA_HOST may point at a single interface that is not the one carrying the traffic. Compare the address in ss -tlnp with the interface in ip -brief addr. [source]
loadFn func(req *LlmRequest, systemInfo ml.SystemInfo, gpus []ml.DeviceInfo, requireFull bool) bool newServerFn func(systemInfo ml.SystemInfo, gpus []ml.DeviceInfo, model string, f *gguf.Model, adapters []string, projectors []string, opts api.Options, numParallel int, config llm.LlamaServerConfig) (llm.LlamaServer, error) getGpuFn func(ctx context.Context, runners []ml.FilteredRunnerDiscovery) []ml.DeviceInfo getSystemInfoFn func() ml.SystemInfo waitForRecovery time.Duration } [source]
config := llamaServerConfigForModel(req.model) config.ContextShift = req.contextShift llama, err = s.newServerFn(systemInfo, loadGpus, req.model.ModelPath, f, req.model.AdapterPaths, req.model.ProjectorPaths, launchOpts, numParallel, config) if err != nil { // some older models are not compatible with newer versions of llama.cpp // show a generalized compatibility error until there is a better way to // check for model compatibility if errors.Is(err, gguf.ErrUnsupported) || strings.Contains(err.Error(), "failed to load model") { err = fmt.Errorf("%v: this model may be incompatible with your ve [source]
The KV cache is quantized at run time, on activations, in blocks of 32 values with one shared scale, with no calibration and no per-tensor choices. Every token you generate attends to every earlier key and value through that rounding, so a longer context means more rounded values in every attention step. That is why the FAQ says the loss is more noticeable at higher context sizes, and higher context sizes are the only reason you are here. A 4-bit cache at 4k tokens and a 4-bit cache at 64k tokens are not the same risk. [source]
Setting OLLAMA_CONTEXT_LENGTH explicitly overrides the tier in either direction. Raising it on a small machine is the quickest way to make a model fail to load, and lowering it on a large one is a legitimate way to keep more models resident at once. [source]
What remains is that the configuration surface is still 26 variables wide, the documentation lags the code by months at a time, and the defaults that are left are still cautious. The context cache defaults to f16 when q8_0 would save memory for free, and it is a global setting that silently degrades to f16 on architectures that cannot take it, so the one memory lever you are told to pull is also the one you cannot verify from the outside. [source]
On linux, after a suspend/resume cycle, sometimes Ollama will fail to discover your NVIDIA GPU, and fallback to running on the CPU. You can workaround this driver bug by reloading the NVIDIA UVM driver with `sudo rmmod nvidia_uvm && sudo modprobe nvidia_uvm` [source]
Pay closer attention to the GPU tier when your requirement sits near its VRAM limit. [source]
One honest limit: Ollama does not publish a queue depth. ollama ps and the /api/ps endpoint report the models that are loaded, not the requests that are waiting. So you measure the queue from the client side, by watching time to first byte, or you count 503 responses at whatever sits in front. [source]
Weights are quantized once, offline, by whoever built the GGUF. The K in q4_K_M marks a scheme that spends more bits on the tensors that matter most, and the error it introduces is fixed at build time, so it can be measured once against the original before anyone runs the model. [source]

Comparisons and alternatives

The same two lines: OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0. The memory arithmetic is identical whether the cache lives in RAM or VRAM, and on a VPS RAM is the whole budget. Confirm the Flash Attention line in the journal shows it enabled, compare ollama ps at the same CONTEXT before and after, and measure tokens per second yourself rather than assuming it went up. [source]
A 27B model at Q4_K_M needs 16.5 GB for its weights and another 2.3 GB of KV cache and recurrent state at a 32K context. On a 24 GB RTX 4090 that leaves about 6 GB spare and generates about 48 tokens per second, within one token of the 47.7 measured on that exact card. This LLM VRAM calculator does that arithmetic for any model, quantization, context length and GPU, which turns the question of how much VRAM to run an LLM into a number instead of a rule of thumb. Original content from computingforgeeks.com - post 168756 [source]
The PROCESSOR column reads 100% GPU when the whole thing fits. A split such as 35%/65% CPU/GPU means part of the model is running on the CPU. The SIZE column includes the KV cache, so it grows when you raise the slot count and reload the model. Raise OLLAMA_NUM_PARALLEL, restart, send one request, and run ollama ps again: that is the memory cost of your change, measured rather than guessed. If that measurement says the model no longer fits, remember that the weights are the other half of the same budget, and moving from fp16 to a q8 or q4 build often frees more VRAM than the slot you were tryi [source]
Three things the formula does not tell you. Models with sliding-window layers, such as the Gemma family, allocate a much smaller cache for those layers, so the formula overestimates them. Some newer architectures also break the formula's inputs: the Gemma 4 example in Ollama's /api/show reference (checked 13 October 2026) lists gemma4.attention.shared_kv_layers as 18, layers that share key and value cache rather than storing their own, and a key_length of 512 for full-attention layers against a key_length_swa of 256 for sliding-window layers, neither of which equals embedding_length / head_cou [source]
The context (KV) cache grows with your context length and, at 32K or more tokens, can use more memory than the model weights themselves. How much it costs depends on the model you run: the memory Qwen 3.6 27B needs is listed against every Mac configuration, so you can size the weights side of the equation before adding the cache on top. Quantizing it from the default f16 down to q8_0 cuts that roughly in half for a perplexity increase that benchmarks put at 0.002 to 0.05, which nobody notices in practice. q4_0 quarters it with a larger, measurable trade-off. We cover the full quality and memor [source]
Ollama blog (2025-09-23): the new engine measures the exact memory a model needs before running instead of estimating it; enabled by default for models on the new engine (gpt-oss, llama4, gemma3, qwen3, mistral-small3.2, embedding models and others), with more migrating. [source]
Ai AI Coding Ai-Agents Alerting Anaconda Android API Architecture Aws AWS Amplify Backup Bash Cheatsheet Cloud Coding Community CookBook Cursor Data Database DeepLearning Dev Devops DGX Spark Digital Detox Docker Documentation Embeddings Filofax Flutter Git Gitea GitHub Go Golang Gpu Grafana Hardware Hermes Hosting Hugo Images Infrastructure Integration JavaScript K8S Knowledge-Management Kubernetes LabelStudio Latex Linux Llama.cpp Llamacpp Llm Logging Logseq Machine Learning Mainroad Markdown Melbourne Microservices Minio Mmdetection Monitoring Node.js NVidia ObjectDetection Observability Ob [source]
Ollama KV cache quantization: q8_0 vs f16 Guides [source]
Blog › Guides › Ollama KV cache quantization: q8_0 vs f16 [source]
Ollama: Batching Requests vs Parallel Execution [source]
If the model weights plus that KV cache do not fit in VRAM, Ollama moves some layers to system RAM and those layers run on the CPU. CPU layers are far slower than GPU layers, so this makes every request slower, including the single request you started with. Raising parallelism can therefore reduce throughput rather than raise it. With a large enough model the weights alone settle the question before any slot arithmetic begins, which is why self-hosting something the size of Kimi K3 is a conversation about how many cards you have rather than how many slots you set. [source]
Servers built for many simultaneous users work differently. They allocate KV cache in small pages on demand and add arriving requests into a batch that is already running, so memory follows actual demand instead of a fixed division. If your goal is a large number of concurrent users on one GPU, that architectural difference matters more than any value of OLLAMA_NUM_PARALLEL. The comparison between Ollama and vLLM is the place to make that call. Do not switch on principle, though: a different server is more to operate, and if your traffic is a few people, the built-in behaviour is the correct a [source]
The reason you are reading this is usually the same. You picked a weight quantization that fits, then raised num_ctx, and the model stopped fitting, or started swapping. The weights did not change size. The KV cache did, because it grows with context length, and at 32k or 64k tokens it can be bigger than the saving you made by picking q4 weights instead of q8. [source]
Yes, on supported hardware. Since October 2025, OLLAMA_FLASH_ATTENTION has been a three-state override rather than an off-by-default switch: leave it unset and Ollama enables flash attention automatically wherever the selected backend and devices support it, set 1 to force it on, or set 0 to force it off. [source]
ToolPiper takes the other approach. It bundles the same llama.cpp engine and runs the same GGUF models, but it decides the KV cache type per model rather than globally: it reads each model's GGUF header, checks the head dimensions against what the Metal backend can actually serve, and uses q8_0 when it is safe and f16 when it is not. You get the memory saving on every model that supports it and a load that works on the ones that do not, with no variable to set and no silent fallback to discover later. Browser access follows a built-in origin allowlist (loopback pages, modelpiper.com and its ow [source]
And the variables above stop being the interface. The tuning they express - context length, keep-alive, parallelism, flash attention - is settings in a native app rather than environment state you export in a shell profile and re-derive on every machine. That is a transition path, not the recommended setup. The honest limitation: ToolPiper is macOS only. If you run Ollama on Linux or Windows, the variables in this reference are how you get there, and they work. [source]
OLLAMA_KV_CACHE_TYPE: Halve Ollama's KV Cache MemoryThe full memory and quality trade-off behind the KV cache variableOLLAMA_ORIGINS=*: Fix the Ollama CORS Error on MacSet OLLAMA_ORIGINS to unblock browser connections to OllamaRun Multiple Ollama Models on Mac at Once: What Fits in 8, 16, 32GBOLLAMA_MAX_LOADED_MODELS and the memory math for running several modelsInstall Ollama on Mac: Setup Guide and the One-App AlternativeGet Ollama running before you tune it [source]
If you have multiple AMD GPUs in your system and want to limit Ollama to use a subset, you can set `ROCR_VISIBLE_DEVICES` to a comma separated list of GPUs. You can see the list of devices with `rocminfo`. If you want to ignore the GPUs and force CPU usage, use an invalid GPU ID (e.g., "-1"). When available, use the `Uuid` to uniquely identify the device instead of numeric value. [source]
Q4_K_M for almost everyone. It uses 0.60 bytes per parameter against Q8_0’s 1.06, so it nearly halves the VRAM, and the quality cost is small. On Qwen3.5 9B, GSM8K accuracy measured 87.6 at Q8_0 and 84.8 at Q4_K_M. The better surprise sits one step smaller: IQ4_XS scored 86.4 on the same test from a file about 8 percent smaller than Q4_K_M, and it also generated faster. Q4_K_M stays the default because it is what Ollama ships, not because it wins every test. Q8 is worth the extra memory only when a task is sensitive to small numerical drift and the card has the room to spare. Below 4-bit, qual [source]
Three settings matter more than the rest. proxy_buffering off keeps streamed tokens flowing to the client instead of collecting them in nginx. proxy_read_timeout 600s covers long generations on large models, which otherwise die at the 60 second default. Rewriting Host to 127.0.0.1:11434 matters because Ollama validates the origin of incoming requests and answers 403 Forbidden when the forwarded hostname is not on its allow list. Alternatively, keep the external hostname and list it in OLLAMA_ORIGINS. [source]
The server reads environment variables once at launch, so a changed OLLAMA_HOST or OLLAMA_MODELS only takes effect after Ollama is restarted. An unparseable port in OLLAMA_HOST logs a warning and silently reverts to the default instead of failing. [source]
Every input is traceable. Weights are the real GGUF file sizes read from the Hugging Face repositories people actually download, not parameters multiplied by a guessed byte count. The KV cache comes from the config.json shipped with each model, which matters because the hybrid attention in Qwen3.8, Gemma 4 and Nemotron cuts the cache four to forty times against an older dense model. Speed comes from a model fitted to 20 single-stream runs on five NVIDIA GPUs, the same runs behind the tested GPU buyer guide and the Ollama vs vLLM vs llama.cpp benchmark, then checked against the public llama.cpp [source]
llama.cpp resolved flash attention to disabled, so it will not allocate a quantized value cache. Set OLLAMA_FLASH_ATTENTION=1 explicitly rather than leaving it on auto, then reload and restart. If the same line comes back with it forced on, the backend for your hardware does not support the attention kernel, and the cache has to stay f16 on that box. [source]
Ollama has quietly gotten better at this. Flash attention turns itself on now, context length scales with your VRAM instead of sitting at 4096, and the loaded-model limit adapts to your GPU count. Those were all manual variables not long ago, and the honest read is that the gap between "Ollama out of the box" and "Ollama tuned" is smaller in v0.32.5 than it has ever been. [source]
For readers who want the table rather than the tool, these are the real GGUF weight files for a current model at each size. Add the KV cache for the context you plan to run (next section) and about 0.7 GB of runtime overhead. Q4_K_M is the column that matters for local use, because it is the default quantization in Ollama and the usual balance of size and quality. [source]
• A 6.5 bit per value KV cache with q8_0 for the K cache and q4_0 for the V cache also seems to be more precise than q6_K weights. There seems to be no significant quality loss from using q8_0 instead of FP16 for the KV cache. [source]
That is Ollama's queue overflow error, returned with HTTP status 503. The number of requests waiting to be scheduled, meaning waiting for their model to be loaded, reached OLLAMA_MAX_QUEUE, which defaults to 512, so the newest request was rejected instead of being added. Requests to a model that is already loaded do not count against that limit. You will mostly see this error during bursts at startup or while models are being swapped. It is not a memory error and it is not a model error. Raising the queue only makes callers wait longer before the same rejection, so the real fixes are more slot [source]
Two caveats the FAQ states but most guides drop. First, this is a global setting: every model the server loads runs with the type you set, and there is no per-model override. A Modelfile parameter for it was proposed in December 2024 and has never been merged, so any guide showing kv_cache_type in a Modelfile or /set parameter is quoting an unmerged pull request. Second, quantized KV cache silently falls back to f16 on model architectures that do not support it, so setting q8_0 is not a guarantee that you got it. Check your memory use rather than assuming. [source]
There is a genuine trap that keeps the myth alive. Ollama's startup environment dump renders this variable's value as false when it is unset, because the dump asks for the value with a false fallback rather than for the resolved runtime decision. So you can start the server, read OLLAMA_FLASH_ATTENTION:false in your own log, and reasonably conclude flash attention is off while it is actively running. That log line reports the variable, not the behavior. [source]
Yes, on hardware that supports it. Since a change on 2025-10-01, OLLAMA_FLASH_ATTENTION is a three-state override: unset means Ollama decides automatically based on the backend and devices, 1 forces it on, and 0 forces it off. Guides telling you to set it to 1 to enable it are describing behavior that ended in October 2025. Ollama's own FAQ was not corrected until July 2026. One thing that keeps the confusion alive: the server's startup log prints OLLAMA_FLASH_ATTENTION:false when the variable is unset, which reports the variable rather than the resolved behavior. [source]
Hostinger provides a preconfigured Ollama template that you can deploy with a few clicks, instead of installing Ollama, GPU drivers, and other dependencies yourself from the Linux command line. [source]
The number that matters is in the 32B row. The RTX 5090 with 32 GB runs the 32B model at 71 tokens per second, more than double the L40S with its larger 48 GB. More VRAM did not make it faster. Token generation is bound by memory bandwidth, not capacity, and the 1,792 GB/s of GDDR7 on the 5090 is more than twice the 864 GB/s of GDDR6 in the L40S. VRAM capacity decides what fits; memory bandwidth decides how fast it runs once it fits. The RTX 3090 vs RTX 5090 comparison shows the same effect across more models. [source]
Adding a GPU significantly speeds up response generation compared with running Ollama on a CPU alone. In our benchmark on a Hostinger GPU instance with an RTX 4090, the Llama 3.1 8B Q4_K_M model generated responses 2.7× faster than on the CPU-only instance. [source]
Running the K/V context cache at Q8_0 quantisation effectively halves the VRAM required for the context compared to the default F16 with minimal quality impact on the generated outputs, while Q4_0 cuts it down to just one third (at the cost of some noticeable quality reduction). [source]
Saving that 3GB of vRAM by using Q8_0 could be enough to either allow you to double the context size to 64K, or perhaps to run a larger parameter model (e.g. 14B instead of 8B). [source]
Q8_0 & Q4_0Different levels of quantisation, with Q8_0 halving the VRAM usage of the context and Q4_0 reducing it to one third compared to F16 (unquantised). [source]
The current Apple lineup spreads that trade-off wide. The M5 Max moves 614 GB/s with up to 128 GB, the M5 Ultra 1,228 GB/s with up to 512 GB, while Strix Halo and DGX Spark both sit near 256 to 273 GB/s with 128 GB. On the same scoreboard model the M5 Max reaches 119.9 tok/s, the M5 Ultra 179.1, the DGX Spark 57.2, and an RTX 4090 about 170. The full comparison of unified memory vs VRAM for AI covers where each class wins. [source]
connection refused from a remote host. The service is still on loopback. ss -tlnp shows 127.0.0.1:11434 rather than *:11434. Re-check the drop-in, run sudo systemctl daemon-reload, and restart the service. A systemctl edit file that was saved without the [Service] header is silently ignored. [source]
Choose a Hostinger GPU based on the model size, context length, and workloads you plan to run. Give yourself some VRAM headroom instead of choosing a GPU that only barely fits your current model. [source]
So here it is: every Ollama environment variable, what it controls, its real default, and where to set it on macOS, Linux, Docker, and Windows. This reference was rebuilt on 2026-07-29 by reading envconfig/config.go at v0.32.5 line by line rather than by paraphrasing the FAQ, because on several variables the two disagree and the source wins. This reference assumes Ollama is already running. If it is not, our install guide for Mac gets you there first. [source]
ROCm 10 OpenCL vs. Mesa 26.3-dev Rusticl Performance On AMD Strix Halo [source]
To change where Ollama stores the downloaded models instead of using your home directory, set the environment variable `OLLAMA_MODELS` in your user account. [source]
A model like gpt-oss-120b activates only 5.1 billion of its 117 billion parameters per token, and DeepSeek V4 Flash activates 13 billion of 284 billion. Those active counts make the model generate faster, but they do nothing for VRAM. The full weight set still has to fit in memory. gpt-oss-120b needs 62.8 GB for weights alone, so it lands on a 96 GB card or a 128 GB unified-memory machine rather than a consumer GPU. Sizing a mixture-of-experts model by its active parameters instead of its total is the most common and most expensive mistake in this whole topic. The reverse also holds: a MoE tha [source]
PreviousBest GPU for LLMs in 2026: Tested by VRAM, Budget, and Model SizeUp next Unified Memory vs VRAM for AI, Explained [source]
GPU: centron vs. HetznerNVIDIA models and hourly prices GPU: centron vs. IONOSPrices, models and scaling GPU: centron vs. OVHcloudA100 and RTX instances [source]
// numCtxAuto is true when NumCtx came from Ollama's automatic VRAM-tier // default rather than explicit request, model, or environment config. numCtxAuto bool [source]
OLLAMA_HOST also has parsing behavior worth knowing: it defaults the scheme to http and the port to 11434, uses https on port 443 for ollama.com, expects IPv6 addresses in brackets, and on an unparseable port logs a warning and silently reverts to the default rather than failing to start. [source]
Start the server with OLLAMA_DEBUG=1 and Ollama logs its environment at startup. The dump is close to complete but not exact: at least one variable is missing from it, and two report a placeholder instead of the value actually in effect. [source]
Now add an 8 GB GPU. The 4.9 GB model fits in the GPU’s own memory, called VRAM, with room for a shorter context, so Ollama uses the GPU for processing instead of relying on the CPU. [source]
Ollama has some KV cache optimizations to improve how memory can be efficiently used. Ollama configures causal attention at the individual model level instead of configuring as a group. [source]
// numBatchAuto is true when NumBatch came from Ollama's default options // rather than an explicit request or model option. numBatchAuto bool [source]

Facts and statements

TL;DR Ollama is configured through environment variables read by the server at startup. The most-used are OLLAMA_HOST (bind address), OLLAMA_MODELS (storage path), OLLAMA_KEEP_ALIVE (how long models stay loaded), and OLLAMA_KV_CACHE_TYPE (cache quantization). OLLAMA_FLASH_ATTENTION is no longer an on/off switch you must set: since October 2025 it is a three-state override, and Ollama enables flash attention automatically wherever the backend and device support it. Set variables in your shell, with launchctl on the macOS app, or systemd on Linux, then restart Ollama. [source]
OLLAMA_NUM_PARALLEL defaults to 1 in envconfig; required memory scales with OLLAMA_NUM_PARALLEL * OLLAMA_CONTEXT_LENGTH (a 2K context with 4 parallel requests becomes 8K). [source]
Default keep-alive is 5 minutes. Per-request keep_alive on /api/generate and /api/chat overrides OLLAMA_KEEP_ALIVE; 0 unloads right after the response, negative (e.g. -1) keeps loaded forever. Env parsing accepts durations or integer seconds; an unparseable value silently keeps 5m. [source]
if runtime.GOOS != "darwin" { ret["CUDA_VISIBLE_DEVICES"] = EnvVar{"CUDA_VISIBLE_DEVICES", CudaVisibleDevices(), "Set which NVIDIA devices are visible"} ret["HIP_VISIBLE_DEVICES"] = EnvVar{"HIP_VISIBLE_DEVICES", HipVisibleDevices(), "Set which AMD devices are visible by numeric ID"} ret["ROCR_VISIBLE_DEVICES"] = EnvVar{"ROCR_VISIBLE_DEVICES", RocrVisibleDevices(), "Set which AMD devices are visible by UUID or numeric ID"} ret["GGML_VK_VISIBLE_DEVICES"] = EnvVar{"GGML_VK_VISIBLE_DEVICES", VkVisibleDevices(), "Set which Vulkan devices are visible by numeric ID"} ret["GPU_DEVICE_ORDINAL"] = EnvVa [source]
Vulkan first shipped as experimental. On 2025-11-14 Phoronix reported ollama 0.12.11 with Vulkan enabled by setting OLLAMA_VULKAN=1, aimed at older AMD GPUs without ROCm support and RADV systems without ROCm installed. [source]
OLLAMA_MAX_LOADED_MODELS: env default is 0, which the scheduler turns into 3 * max(GPU count,1); a model loads concurrently only if it fits in memory, and on GPU must fit fully in VRAM. [source]
The scheduler forces numParallel to 1 for embedding models and for some architectures (e.g. mllama, qwen35, lfm2), regardless of OLLAMA_NUM_PARALLEL. [source]
NVIDIA support needs compute capability 5.0+ and driver 550 or newer. Cards with compute capability 5.0 through 6.2 need driver 570 or newer. [source]
Without HSA_OVERRIDE_GFX_VERSION, an unsupported gfx1032 card (RX 6650 XT) crashed in rocBLAS with a missing TensileLibrary.dat and a segmentation fault. Setting 10.3.0 in the docker run command fixed it (ollama/ollama:0.1.27-rocm, March 2024). [source]
实际配置时，先用 ollama ps 看模型加载位置，再用 nvidia-smi 或 ROCm 工具观察显存占用。需要限制 GPU 时，NVIDIA 用 CUDA_VISIBLE_DEVICES，AMD ROCm 用 ROCR_VISIBLE_DEVICES，Vulkan 用 GGML_VK_VISIBLE_DEVICES。如果在 Docker 中运行，先确保容器层面能看到 GPU。 [source]
Required memory scales with OLLAMA_NUM_PARALLEL x context length: a 2K context with 4 parallel requests becomes an 8K context allocation. Flash attention reduces memory as context grows; quantized KV cache needs flash attention (q8_0 about 1/2 of f16, q4_0 about 1/4). [source]
Requests for an already-loaded model reuse the runner and skip the pending queue, so the 503 applies to requests needing a model load, not those waiting on a parallel slot. [source]
On Windows with Radeon GPUs the FAQ says max loaded models defaults to 1 due to ROCm v5.7 VRAM reporting limits (note may be outdated). [source]
On Linux, Ollama requires the AMD ROCm v7 driver. On Windows it requires a ROCm v7 / HIP7-capable driver stack. The Windows supported-card list is shorter than the Linux list (Radeon RX 7600 through 7900 XTX and Radeon PRO W7500 through W7900). [source]
OLLAMA_GPU_OVERHEAD reserves a number of bytes of VRAM per GPU (default 0), shrinking what the scheduler treats as available. [source]
Hardware support is still expanding, which is why the honest answer needs a hardware qualifier and not just a version. v0.31.2 enabled flash attention on NVIDIA GPUs with compute capability 6.x after upstream Pascal kernel fixes, and the gate requires every enumerated device to qualify, so one unsupported GPU in a multi-GPU box turns it off for all of them. Setting OLLAMA_FLASH_ATTENTION=1 is still useful as a forcing function, and 0 is the documented way to rule it out while debugging. [source]
HSA_OVERRIDE_GFX_VERSION=x.y.z forces ROCm to use a nearby supported LLVM target for an unsupported AMD GPU, for example 10.3.0 for a gfx1032 or gfx1034 card. With mixed GPUs, append the device number (HSA_OVERRIDE_GFX_VERSION_0, HSA_OVERRIDE_GFX_VERSION_1). The documented override is Linux-only. [source]
With SELinux, containers may be denied access to AMD GPU devices. Run `sudo setsebool container_use_devices=1` on the host. [source]
Verify the settings on Linux with journalctl -u ollama: the 'llama_kv_cache: size' line shows K (q8_0) / V (q8_0), the 'starting llama-server' line shows --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on, and the startup config shows OLLAMA_KV_CACHE_TYPE:q8_0. ollama ps shows SIZE and CONTEXT. [source]
Current envconfig exposes llama.cpp fit controls: LLAMA_ARG_FIT (auto-fit unset memory options, default on) and LLAMA_ARG_FIT_TARGET (free VRAM margin per device, MiB). [source]
docker run --rm -e OLLAMA_NUM_PARALLEL=2 -p 11434:11434 ollama/ollama [source]
<Note> Some RDNA2 / Radeon RX 6000 systems, including RX 6800-class cards, may not expose ROCm v7 on current Windows AMD drivers. Vulkan is enabled by default and is the recommended fallback for those systems. If a mixed iGPU/dGPU system selects an unstable Vulkan iGPU, set `GGML_VK_VISIBLE_DEVICES` to the discrete GPU index. </Note> [source]
Ollama v0.32.5 reads 26 OLLAMA_ variables in envconfig/config.go, plus the GPU selection variables (CUDA_VISIBLE_DEVICES, HIP_VISIBLE_DEVICES, ROCR_VISIBLE_DEVICES, GGML_VK_VISIBLE_DEVICES, GPU_DEVICE_ORDINAL, HSA_OVERRIDE_GFX_VERSION) and the standard proxy variables it inherits from the environment. The official FAQ documents roughly half of them. [source]
On systemd installs the variables must be set as Environment= lines on the ollama unit, followed by daemon-reload and restart. Exporting them in a shell does not affect the managed server. A restart unloads all models. [source]
export OLLAMA_MAX_LOADED_MODELS=2 export OLLAMA_NUM_PARALLEL=4 [source]
OLLAMA_NUM_PARALLEL controls how many requests Ollama will execute in parallel. If you send multiple requests to the same Ollama server, this setting largely decides whether they run concurrently or queue up. [source]
export OLLAMA_NUM_PARALLEL=2 ollama serve [source]
If you’re optimizing a specific page for CLI usage, see the Ollama CLI section in the cheatsheet, plus command examples for ollama serve, ollama ps, and ollama run. [source]
The serving framework matters here more than the model. Ollama and llama.cpp pre-allocate a full-context KV cache for every parallel slot they open (OLLAMA_NUM_PARALLEL, or -np on llama-server), so each extra slot costs a whole cache up front whether it is used or not. vLLM uses PagedAttention, which allocates the cache in small pages on demand and wastes under 4 percent of it, against the large fraction a pre-allocating runtime leaves idle. That efficiency is why vLLM serves far more concurrent requests in the same VRAM and is the right tool once more than one person is hitting the endpoint. [source]
Raised num_ctx and Ollama no longer fits? The KV cache grew, not the weights. Set OLLAMA_KV_CACHE_TYPE with flash attention and watch the change in ollama ps. [source]
OLLAMA_KV_CACHE_TYPE sets the K/V cache type. Valid values are f16 (default), q8_0 and q4_0. [source]
Default multi-GPU placement: if the model fits entirely on any single GPU, Ollama loads it on that GPU (less PCIe traffic); only if it does not fit on one GPU is it spread across all available GPUs. [source]
$ systemctl show ollama.service -p Environment Environment=OLLAMA_HOST=0.0.0.0:11434 OLLAMA_ORIGINS=https://chat.example.com [source]
var ( // NumParallel sets the number of parallel model requests. NumParallel can be configured via the OLLAMA_NUM_PARALLEL environment variable. NumParallel = Uint("OLLAMA_NUM_PARALLEL", 1) // MaxRunners sets the maximum number of loaded models. MaxRunners can be configured via the OLLAMA_MAX_LOADED_MODELS environment variable. MaxRunners = Uint("OLLAMA_MAX_LOADED_MODELS", 0) // MaxQueue sets the maximum number of queued requests. MaxQueue can be configured via the OLLAMA_MAX_QUEUE environment variable. MaxQueue = Uint("OLLAMA_MAX_QUEUE", 512) // MaxTransferStreams caps the number of simultane [source]
OLLAMA_NUM_PARALLEL environment variable [source]
Symptoms that OLLAMA_NUM_PARALLEL is too high [source]
Symptoms that OLLAMA_NUM_PARALLEL is too low [source]
OLLAMA_NUM_PARALLEL=2 (or higher if you have headroom) [source]
CudaVisibleDevices = String("CUDA_VISIBLE_DEVICES") HipVisibleDevices = String("HIP_VISIBLE_DEVICES") RocrVisibleDevices = String("ROCR_VISIBLE_DEVICES") VkVisibleDevices = String("GGML_VK_VISIBLE_DEVICES") GpuDeviceOrdinal = String("GPU_DEVICE_ORDINAL") HsaOverrideGfxVersion = String("HSA_OVERRIDE_GFX_VERSION") ) [source]
解释 llama-bench 的 pp、tg、量化和 Flash Attention 指标，给出可复现的 CUDA、ROCm、Vulkan 测试命令，并说明不同社区跑分为何不能直接组成显卡天梯。 [source]
Ollama supports NVIDIA GPUs with CUDA compute capability 5.0 or higher. This includes many GeForce GTX and RTX cards, as well as data center GPUs such as the A100, H100, and H200. [source]
GPU OOM / model unloading happens frequently [source]
Ollama's HTTP API has no authentication by default; binding OLLAMA_HOST to 0.0.0.0 lets anyone who can reach the port use it. Guard it with a firewall or a reverse proxy such as Nginx (the FAQ shows a proxy config). [source]
// Establish a baseline before we unload gpusBefore := s.getGpuFn(context.Background(), runners) var totalMemoryBefore, freeMemoryBefore uint64 for _, gpu := range gpusBefore { totalMemoryBefore += gpu.TotalMemory freeMemoryBefore += gpu.FreeMemory } totalMemoryNow := totalMemoryBefore freeMemoryNow := freeMemoryBefore [source]
OLLAMA_ORIGINS controls which browser origins may call the API and is separate from OLLAMA_HOST. Defaults allow 127.0.0.1 and 0.0.0.0 origins; extra origins such as chrome-extension://* must be added explicitly. [source]
Per the Ollama FAQ, q8_0 uses approximately 1/2 the memory of f16 with a very small precision loss (recommended if not using f16). q4_0 uses approximately 1/4 the memory with a small-to-medium loss that may be more noticeable at higher context sizes. [source]
On constrained hardware, Ollama may try to allocate a vision model's full advertised context and crash. OpenClaw's docs advise capping num_ctx in model params (example 2048) and, optionally, a short keep_alive. [source]
```bash theme={"system"} launchctl setenv OLLAMA_HOST "0.0.0.0:11434" ``` [source]
1. Edit the systemd service by calling `systemctl edit ollama.service`. This will open an editor. [source]
Note: Windows with Radeon GPUs currently default to 1 model maximum due to limitations in ROCm v5.7 for available VRAM reporting. Once ROCm v6.2 is available, Windows Radeon will follow the defaults above. You may enable concurrent model loads on Radeon on Windows, but ensure you don't load more models than will fit into your GPU's VRAM. [source]
Higher values can increase throughput if you have enough CPU/GPU/VRAM, but may increase latency and memory pressure. [source]
Start with 1–2 for a single GPU / limited VRAM, then increase gradually while watching: [source]
The data behind this chart [ { "label": "Weights, q4_K_M", "gib": 4.56 }, { "label": "Weights, q8_0", "gib": 7.92 }, { "label": "KV cache f16, num_ctx 32768", "gib": 4.0 }, { "label": "KV cache f16, num_ctx 65536", "gib": 8.0 } ] [source]
• AMD: Works on most AMD GPUs with ROCm support, although ROCm in general is not as well supported as CUDA or Metal and performance may vary. [source]
// On iGPUs, GPU free memory can be a static or slowly refreshed device // baseline. updateFreeSpace has already subtracted known Ollama runner // allocations from that baseline. Current system free memory is a separate // live measurement that already includes those loaded runners, so use the // smaller value for shared-memory GPUs without discounting discrete VRAM. if systemInfo.FreeMemory > 0 && sharedGPUFree > 0 && systemInfo.FreeMemory < sharedGPUFree { return discreteGPUFree + systemInfo.FreeMemory, gpuFree, true } [source]
An incoming request has three possible fates. It starts immediately in a free slot. It waits in the queue. Or the queue is already full and the server refuses it with HTTP 503. Which one you get is decided by OLLAMA_NUM_PARALLEL, OLLAMA_MAX_QUEUE and OLLAMA_MAX_LOADED_MODELS. There is one catch in current releases as of October 2026. The 503 only applies to requests that arrive while their model still has to be loaded. Ollama never refuses a request for a model that is already loaded. That request waits for a slot for as long as it takes. [source]
#ollama#kv-cache#quantization#num_ctx#ram#flash-attention [source]
想限制 Ollama 使用哪些 GPU，需要用 CUDA_VISIBLE_DEVICES、ROCR_VISIBLE_DEVICES 或 GGML_VK_VISIBLE_DEVICES。 [source]
llama.cpp GPU 性能怎么比：CUDA、ROCm、Vulkan 复测与选卡方法 [source]
curl -s http://127.0.0.1:11434/api/generate -d '{ "model": "llama3.1:8b-instruct-q4_K_M", "prompt": "Explain how DNS resolution works in at least 300 words.", "stream": false, "keep_alive": "10m", "options": { "num_ctx": 8192, "num_predict": 128, "temperature": 0, "seed": 42, "num_gpu": 0 } }' | python3 -c 'import sys,json; d=json.load(sys.stdin); print("{:.2f} tokens/s".format(d["eval_count"]/(d["eval_duration"]/1e9)))' [source]
curl -s http://127.0.0.1:11434/api/generate -d '{ "model": "llama3.1:8b-instruct-q4_K_M", "prompt": "Explain how DNS resolution works in at least 300 words.", "stream": false, "keep_alive": "10m", "options": { "num_ctx": 8192, "num_predict": 128, "temperature": 0, "seed": 42, "num_gpu": -1 } }' | python3 -c 'import sys,json; d=json.load(sys.stdin); print("{:.2f} tokens/s".format(d["eval_count"]/(d["eval_duration"]/1e9)))' [source]
graph TD A["Client or app"] -->|"HTTPS + Basic Auth"| B["nginx :443"] B -->|"HTTP to 127.0.0.1:11434"| C["ollama serve"] C --> D["GPU model runner"] E["Public internet"] -.->|"dropped by firewall"| C [source]
for _, gpu := range loadGpus { available := gpu.FreeMemory - envconfig.GpuOverhead() - gpu.MinimumMemory() if gpu.FreeMemory < envconfig.GpuOverhead()+gpu.MinimumMemory() { available = 0 } slog.Info("gpu memory", "id", gpu.ID, "library", gpu.Library, "available", format.HumanBytes2(available), "free", format.HumanBytes2(gpu.FreeMemory), "minimum", format.HumanBytes2(gpu.MinimumMemory()), "overhead", format.HumanBytes2(envconfig.GpuOverhead())) } [source]
func constrainedCUDAWithoutFlashAttention(effectiveCtx int, gpus []ml.DeviceInfo) bool { if effectiveCtx <= 4096 { return false } return slices.ContainsFunc(gpus, func(gpu ml.DeviceInfo) bool { if gpu.Library != "CUDA" { return false } memory := gpu.FreeMemory if memory == 0 || (gpu.TotalMemory > 0 && gpu.TotalMemory < memory) { memory = gpu.TotalMemory } return memory > 0 && memory <= 8*format.GibiByte }) } [source]
// Don't reload runner if num_gpu=-1 was provided optsExisting := runner.Options.Runner optsNew := req.opts.Runner optsNew.NumCtx = effectiveContext(optsNew.NumCtx, runner.trainContext) if runner.numCtxAuto && req.numCtxAuto { optsNew.NumCtx = optsExisting.NumCtx } if runner.numBatchAuto && req.numBatchAuto { optsNew.NumBatch = optsExisting.NumBatch } if runner.useMMapAuto && optsNew.UseMMap == nil { optsNew.UseMMap = optsExisting.UseMMap } if optsNew.NumGPU < 0 { optsExisting.NumGPU = -1 optsNew.NumGPU = -1 } [source]
SIZE is still far bigger than the weights. Read -c and -np on the starting llama-server line. If -c is a multiple of your num_ctx, you have more than one parallel slot, and every slot gets a full cache. Also check that CONTEXT in ollama ps is what you asked for; a client that sets its own num_ctx overrides yours, and how Ollama picks the context length has more than one input. [source]
docker run -e OLLAMA_KV_CACHE_TYPE=q8_0 -p 11434:11434 ollama/ollama [source]
4. Edit or create a new variable for your user account for `OLLAMA_HOST`, `OLLAMA_MODELS`, etc. [source]
The `keep_alive` API parameter with the `/api/generate` and `/api/chat` API endpoints will override the `OLLAMA_KEEP_ALIVE` setting. [source]
* Linux Intel GPU Instructions - [https://dgpu-docs.intel.com/driver/client/overview.html](https://dgpu-docs.intel.com/driver/client/overview.html) * Linux AMD GPU Instructions - [https://amdgpu-install.readthedocs.io/en/latest/install-script.html#specifying-a-vulkan-implementation](https://amdgpu-install.readthedocs.io/en/latest/install-script.html#specifying-a-vulkan-implementation) [source]
Weight bytes per parameter (Llama 3.3 70B): BF16 2.00, Q8_0 1.06, Q5_K_M 0.71, Q4_K_M 0.60, Q4_0 0.57, IQ4_XS 0.54. Pre-quantized models such as gpt-oss-120b (MXFP4) break this rate, so read the file size instead. [source]
• Set the K/V cache quantisation to Q8_0 by adding OLLAMA_KV_CACHE_TYPE="q8_0" to the environment variables you run Ollama with. [source]
Browser origins systemctl show ollama.service -p Environment OLLAMA_ORIGINS lists known hosts only [source]
The second request waits, or it is refused: what OLLAMA_NUM_PARALLEL and OLLAMA_MAX_QUEUE really control, and why every parallel slot costs you memory. [source]
What OLLAMA_NUM_PARALLEL, OLLAMA_MAX_QUEUE and OLLAMA_MAX_LOADED_MODELS each control [source]
Copy [Service] Environment="OLLAMA_NUM_PARALLEL=4" Environment="OLLAMA_MAX_LOADED_MODELS=1" Environment="OLLAMA_MAX_QUEUE=32" [source]
Copy [Service] Environment="OLLAMA_FLASH_ATTENTION=1" Environment="OLLAMA_KV_CACHE_TYPE=q8_0" [source]
// Set aside VRAM per GPU var GpuOverhead = Uint64("OLLAMA_GPU_OVERHEAD", 0) [source]
{ modelfile: '# Modelfile generated by "ollama show"\n# To build a new Modelfile based on this one, replace the FROM line with:\n# FROM llava:latest\n\nFROM ~/.ollama/models/blobs/sha256:200765e1283640ffbd013184bf496e261032fa75b99498a9613be4e94d63ad52\nTEMPLATE """{{ .System }}\nUSER: {{ .Prompt }}\nASSISTANT: """\nPARAMETER num_ctx 4096\nPARAMETER stop "\u003c/s\u003e"\nPARAMETER stop "USER:"\nPARAMETER stop "ASSISTANT:"', parameters: 'num_keep 24\nstop "<|start_header_id|>"\nstop "<|end_header_id|>"\nstop "<|eot_id|>"', template: "{{ if .System }}<|start_header_id|>system<|end_head [source]
整理 Ollama 多显卡使用中的关键信息：什么时候会跨 GPU 加载模型、如何用 CUDA_VISIBLE_DEVICES / ROCR_VISIBLE_DEVICES 限制显卡、显存是否能叠加、不同 GPU 能不能混用，以及 Docker、PCIe 带宽和性能方面的常见误区。 [source]
OLLAMA_VULKAN=1 GGML_VK_VISIBLE_DEVICES=0 ollama serve [source]
老卡要先确认 CUDA compute capability 或 ROCm 支持。 [source]
launchctl setenv OLLAMA_KV_CACHE_TYPE q8_0 [source]
KV cache typef16, global across every model, set OLLAMA_KV_CACHE_TYPE to changeDecided per model from the GGUF header: q8_0 where the head dimensions allow it, f16 where they do not [source]
For the macOS app, run launchctl setenv OLLAMA_KV_CACHE_TYPE q8_0. For a terminal server, add export OLLAMA_KV_CACHE_TYPE=q8_0 to ~/.zshrc. Substitute whichever variable from the reference tables you need. [source]
The right GPU also depends on Ollama’s hardware compatibility, including NVIDIA CUDA, AMD ROCm, Apple Silicon, and Vulkan-supported devices. [source]
Ollama supports NVIDIA GPUs, selected AMD GPUs, and Apple Silicon Macs, as well as additional GPUs on Windows and Linux through Vulkan. [source]
Ollama supports Vulkan on Windows and Linux, giving it access to GPUs beyond those supported by NVIDIA CUDA and AMD ROCm. [source]
Performance through Vulkan is generally slower than CUDA or ROCm, but still faster than CPU-only. [source]
Ollama binds 127.0.0.1:11434 by default; OLLAMA_HOST changes the bind address (e.g. 0.0.0.0:11434 for all interfaces, or a single private IP). The API has no built-in authentication or rate limiting, so exposing it needs a firewall source-subnet rule or a reverse proxy with auth and TLS. [source]
if runner != nil { if runner.needsReload(ctx, pending) { slog.Debug("reloading", "runner", runner) runnerToExpire = runner } else { // Runner is usable, return it logutil.Trace("using existing loaded runner", "model", pendingKey) pending.useLoadedRunner(runner, s.finishedReqCh) break } } else if maxRunners > 0 && loadedCount >= int(maxRunners) { slog.Debug("max runners achieved, unloading one to make room", "runner_count", loadedCount) runnerToExpire = s.findRunnerToUnload() } else { // Either no models are loaded or below envconfig.MaxRunners // Get a refreshed GPU list var gpus []ml.DeviceIn [source]
Ollama binds 127.0.0.1 port 11434 by default; the bind address is changed with the OLLAMA_HOST environment variable (for example 0.0.0.0:11434 to listen on all interfaces). [source]
The Windows installer does not need Administrator, installs to the home directory by default, and accepts `OllamaSetup.exe /DIR="d:\some\location"` to relocate binaries. Models are stored separately via OLLAMA_MODELS, and uninstalling does not remove models stored at a custom OLLAMA_MODELS location. [source]
In that report, setting OLLAMA_CONTEXT_LENGTH had no effect. The working path was `/set parameter num_ctx N` in the REPL followed by `/save <name>`, since `/set` does not persist after exit. [source]
If Ollama is run as a systemd service, environment variables should be set using `systemctl`: [source]
```ini theme={"system"} [Service] Environment="OLLAMA_HOST=0.0.0.0:11434" ``` [source]
* `f16` - high precision and memory usage (default). * `q8_0` - 8-bit quantization, uses approximately 1/2 the memory of `f16` with a very small loss in precision, this usually has no noticeable impact on the model's quality (recommended if not using f16). * `q4_0` - 4-bit quantization, uses approximately 1/4 the memory of `f16` with a small-medium loss in precision that may be more noticeable at higher context sizes. [source]
For AMD GPUs on some Linux distributions, you may need to add the `ollama` user to the `render` group. [source]
// evictAllAndWait synchronously expires every currently loaded runner except // the one being loaded (matched by modelKey) and waits for all unload events // to drain. Returns false if the context was cancelled mid-wait so the caller // can exit the scheduling loop. Used by the OOM retry path in processPending. func (s *Scheduler) evictAllAndWait(ctx context.Context, keepKey string) bool { s.loadedMu.Lock() runnersToExpire := make([]*runnerRef, 0, len(s.loaded)) for key, r := range s.loaded { if key == keepKey { continue } runnersToExpire = append(runnersToExpire, r) } s.loadedMu.Unlock() [source]
Moving the weights from q8_0 (7.92 GiB) to q4_K_M (4.56 GiB) saved about 3.4 GiB. Moving num_ctx from 32768 to 65536 at f16 took the cache from 4.0 GiB to 8.0 GiB, an extra 4 GiB. The context step cost more than the weight step saved. That is the whole trap, and it is why working out whether a model fits in your RAM has to include the context you intend to run as well as the file size on the library page. [source]
The documented manual systemd setup creates a system user and group `ollama` (home /usr/share/ollama, shell /bin/false), adds the current user to that group, and writes /etc/systemd/system/ollama.service with ExecStart=/usr/bin/ollama serve, User=ollama, Restart=always, RestartSec=3, WantedBy=multi-user.target; then daemon-reload and enable/start. Logs are read with `journalctl -e -u ollama`. [source]
* `OLLAMA_KV_CACHE_TYPE` - The quantization type for the K/V cache. Default is `f16`. [source]
* Windows 10 22H2 or newer, Home or Pro * NVIDIA 551.61 or newer Drivers if you have an NVIDIA card * AMD ROCm v7 / HIP7-capable driver stack for ROCm acceleration, or a Vulkan-capable AMD Radeon driver for Vulkan acceleration [source]
Llama 2 7B Q4_0, RTX 5090290 tok/sabout 298 tok/sllama.cpp CUDA scoreboard [source]
Understand Ollama concurrency, queueing, and how to tune OLLAMA_NUM_PARALLEL for stable parallel requests. [source]
This guide explains how Ollama handles parallel requests (concurrency, queuing, and resource limits), and how to tune it using the OLLAMA_NUM_PARALLEL environment variable (and related knobs). [source]
Queuing: If the number of concurrent requests exceeds the configured parallelism (e.g., more than OLLAMA_NUM_PARALLEL requests for a model), additional requests are queued. The queue operates in a first-in, first-out (FIFO) manner. [source]
OLLAMA_NUM_PARALLEL Parallel requests per loaded model 4 (or 1 if limited) [source]
[Service] Environment="OLLAMA_HOST=0.0.0.0:11434" Environment="OLLAMA_ORIGINS=https://chat.example.com" [source]
s.loadedMu.Lock() slog.Debug("got lock to unload expired event", "runner", runner) runnerToUnload := s.loaded[runner.modelKey] if runnerToUnload == nil { // If runnerToUnload is nil, we already processed an event and // unloaded it. This double unload can happen if the initial // request is canceled and we're trying to load another model // that requires this one to be evicted, or the settings change // and require a reload s.loadedMu.Unlock() runner.refMu.Unlock() slog.Debug("duplicate expired event, ignoring", "runner", runner) } else if runner.pid != runnerToUnload.pid { // If the pids do n [source]
Context length and slot count multiply, so they have to be chosen together. A large context with four slots is four large contexts. If you are also tuning the num_ctx context window for your model, change one of the two at a time, or you will not know which one filled the card. [source]
Save and quit. If you prefer a file to an editor, the same three lines in /etc/systemd/system/ollama.service.d/override.conf do the same job, which is what systemctl edit writes for you. Then reload and restart: [source]
Editing "/etc/systemd/system/ollama.service.d/override.conf" canceled: temporary file is empty. You typed the lines below the marker that says everything after it is discarded. Run systemctl edit again and put them above it, or write the override file directly. [source]
AMD ROCm 环境下，用 ROCR_VISIBLE_DEVICES 控制可见 GPU： [source]
Ollama 官方 GPU 文档还提到，如果用实验性的 Vulkan 支持，可以通过 GGML_VK_VISIBLE_DEVICES 选择 Vulkan GPU： [source]
Linux (systemd service): edit the unit with sudo systemctl edit ollama.service and add the variable under [Service]. [source]
Ollama requires the AMD ROCm v7 driver on Linux. You can install or upgrade using the `amdgpu-install` utility from [AMD's ROCm documentation](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/). [source]
Ollama leverages the AMD ROCm library, which does not support all AMD GPUs. In some cases you can force the system to try to use a similar LLVM target that is close. For example The Radeon RX 5400 is `gfx1034` (also known as 10.3.4) however, ROCm does not currently support this target. The closest support is `gfx1030`. You can use the environment variable `HSA_OVERRIDE_GFX_VERSION` with `x.y.z` syntax. So for example, to force the system to run on the RX 5400, you would set `HSA_OVERRIDE_GFX_VERSION="10.3.0"` as an environment variable for the server. If you have an unsupported AMD GPU you can [source]
The Ollama scheduler leverages available VRAM data reported by the GPU libraries to make optimal scheduling decisions. Vulkan requires additional capabilities or running as root to expose this available VRAM data. If neither root access or this capability are granted, Ollama will use approximate sizes of the models to make best effort scheduling decisions. [source]
Choose more VRAM if you plan to use higher-bit quantization, a context longer than 8K, or concurrent requests. [source]
On Windows, OllamaSetup.exe installs per-user without Administrator rights (default under %LOCALAPPDATA%\Programs\Ollama, PATH updated for the user); `OllamaSetup.exe /DIR="d:\some\location"` relocates the binaries. Requirements: Windows 10 22H2 or newer and NVIDIA driver 551.61+ for NVIDIA cards. [source]
// load creates a new model based on req and loads it. If requireFull is true then the model must be loaded fully onto GPUs // (if any). Returns whether the scheduler needs to evict a model to make this one fit. func (s *Scheduler) load(req *LlmRequest, systemInfo ml.SystemInfo, gpus []ml.DeviceInfo, requireFull bool) bool { numParallel := max(int(envconfig.NumParallel()), 1) completion := req.model.CheckCapabilities(model.CapabilityCompletion) == nil [source]
Copy curl -s http://127.0.0.1:11434/api/generate \ -d '{"model":"llama3.1:8b","prompt":"Say hello.","stream":false,"options":{"num_ctx":32768}}' > /dev/null ollama ps sudo journalctl -u ollama --since "5 min ago" --no-pager | grep -E 'starting llama-server|llama_kv_cache: size' [source]
Sick I was able to figure it out. I found this comment ROCm/Tensile#1936 which suggested setting the environment variable HSA_OVERRIDE_GFX_VERSION=10.3.0. That worked for me. The full command I ran to get it to work is: docker run -d -e HSA_OVERRIDE_GFX_VERSION=10.3.0 --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:0.1.27-rocm [source]
docker run -d \ --gpus=all \ -e CUDA_VISIBLE_DEVICES=0,1 \ -v ollama:/root/.ollama \ -p 11434:11434 \ --name ollama \ ollama/ollama [source]
Ollama requires NVIDIA driver version 550 or newer. For GPUs with compute capability 5.0-6.2, driver version 570 or later is required. [source]
Ollama supports GPU acceleration on Apple Silicon Macs through Metal, so you don’t need to install a separate GPU driver. [source]
If Ollama is run as a macOS application, environment variables should be set using `launchctl`: [source]
1. For each environment variable, call `launchctl setenv`. [source]
If you're using the API, use the `keep_alive` parameter with the `/api/generate` and `/api/chat` endpoints to set the amount of time that a model stays in memory. The `keep_alive` parameter can be set to: [source]
slog.Info("evicting all other loaded models for OOM retry", "count", len(runnersToExpire)) for _, runner := range runnersToExpire { runner.refMu.Lock() if runner.expireTimer != nil { runner.expireTimer.Stop() runner.expireTimer = nil } runner.sessionDuration = 0 if runner.refCount <= 0 { s.expiredCh <- runner } runner.refMu.Unlock() } [source]
不过它不是万能开关。开启后并不等于 token/s 线性增长，也可能因为多个模型同时加载、显存估算、上下文长度和 KV cache 增长导致 OOM。官方 FAQ 的核心策略仍然是：如果单 GPU 能完整容纳模型，单 GPU 通常更高效；单 GPU 放不下时才跨多 GPU。 [source]
If Ollama initially works on the GPU in a docker container, but then switches to running on CPU after some period of time with errors in the server log reporting GPU discovery failures, this can be resolved by disabling systemd cgroup management in Docker. Edit `/etc/docker/daemon.json` on the host and add `"exec-opts": ["native.cgroupdriver=cgroupfs"]` to the docker configuration. [source]
A drop-in saved without the `[Service]` header is silently ignored, and a variable exported in a user shell never reaches the systemd-run daemon (the CLI is only an HTTP client). Verify with `systemctl show ollama.service -p Environment` and `ss -tlnp | grep 11434`. [source]
On Windows, Ollama inherits user and system environment variables. Set OLLAMA_HOST, OLLAMA_MODELS etc. under 'Edit environment variables for your account', then quit the tray application and relaunch it from the Start menu (or open a new terminal). If OLLAMA_MODELS was changed, the uninstaller does not remove downloaded models. [source]
Ollama allows cross-origin requests from `127.0.0.1` and `0.0.0.0` by default. Additional origins can be configured with `OLLAMA_ORIGINS`. [source]
• Make sure you're running Ollama with Flash Attention enabled (OLLAMA_FLASH_ATTENTION=1), Note: This should become the default behaviour in the near future as there's no reason not to use it. [source]
It's been noted in llama.cpp's test back at the start of 2024 that high attention head models can be more sensitive to K/V quantisation, so you may want to avoid using K/V context cache quantisation with these models. How high is too high? I'm not sure - but Qwen 2 was mentioned as an example in the llama.cpp PR, I personally have been using Qwen 2.5 Coder 32b Q6_K with Q8_0 K/V cache quantisation without any noticeable issues. [source]
• NVIDIA GPUs: Works on all NVIDIA GPUs with CUDA support, Pascal and newer. [source]
• Using q4_0 for the V cache and FP16 for everything else is more precise than using q6_K with FP16 KV cache. [source]
<Note> While AMD has contributed the `amdgpu` driver upstream to the official linux kernel source, the version is older and may not support all ROCm features. We recommend you install the latest driver from [https://www.amd.com/en/support/download/linux-drivers.html](https://www.amd.com/en/support/download/linux-drivers.html) for best support of your Radeon GPU. </Note> [source]
Llama 2 7B Q4_0, Strix Halo52.5 tok/sabout 52 tok/sllama.cpp Vulkan scoreboard [source]
Llama 2 7B Q4_0, DGX Spark55.8 tok/s57.2 tok/sllama.cpp CUDA scoreboard [source]
The last four rows were never used for fitting. They come from the community scoreboards for CUDA, Vulkan and Apple silicon, and the same 0.82 factor lands within 3 percent on Strix Halo, the M5 Max and the GB10. Three hardware classes needed their own factor, and the calculator applies it automatically: [source]
Model Loading: The number of different models that can be loaded at the same time is controlled by OLLAMA_MAX_LOADED_MODELS. If a request requires loading a new model and memory is insufficient, Ollama will unload idle models to make room, and the request will be queued until the model is loaded. [source]
// loadedModelsForName returns models still owned by loaded runners. The models // are read-only after resolution, so callers may use them after releasing // loadedMu even if a runner unloads in the meantime. func (s *Scheduler) loadedModelsForName(name model.Name, requestedRunner string) []*Model { s.loadedMu.Lock() runners := make([]*runnerRef, 0, len(s.loaded)) for _, runner := range s.loaded { runners = append(runners, runner) } s.loadedMu.Unlock() [source]
slog.Info("selecting GPU backend for llama-server model", "library", selected[0].Library, "gpu_count", len(selected), "available_gpu_count", len(all)) } [source]
llama llm.LlamaServer pid int loading bool // True only during initial load, then false forever gpus []ml.DeviceID // Recorded at time of provisioning discreteGPUs bool // True if all devices are discrete GPUs - used to skip VRAM recovery check for iGPUs runner string vramSize uint64 totalSize uint64 [source]
// CPU, Metal and iGPUs don't need checking, so no waiting required if len(runner.gpus) == 0 || !runner.discreteGPUs || (len(runner.gpus) == 1 && runner.gpus[0].Library == "Metal") { finished <- struct{}{} slog.Debug("no need to wait for VRAM recovery", "runner", runner) return finished } start := time.Now() [source]
// findRunnerToUnload finds a runner to unload to make room for a new model func (s *Scheduler) findRunnerToUnload() *runnerRef { s.loadedMu.Lock() runnerList := make([]*runnerRef, 0, len(s.loaded)) for _, r := range s.loaded { runnerList = append(runnerList, r) } s.loadedMu.Unlock() if len(runnerList) == 0 { slog.Debug("no loaded runner to unload") return nil } [source]
// loadedModels returns a snapshot of the currently loaded models for status // reporting without exposing the scheduler's internal runner bookkeeping. func (s *Scheduler) loadedModels() []loadedModel { s.loadedMu.Lock() runners := make([]*runnerRef, 0, len(s.loaded)) for _, r := range s.loaded { runners = append(runners, r) } s.loadedMu.Unlock() [source]
Blog › Guides › Ollama concurrency: NUM_PARALLEL and MAX_QUEUE [source]
Ollama concurrency: NUM_PARALLEL and MAX_QUEUE [source]
None of this affects the weights. If the model was wrong with an f16 cache, a different cache type will not fix it, and if it fits only with a q4_0 cache the honest options are a smaller weight file, a smaller num_ctx, more RAM, or a different model. [source]
Does OLLAMA_KV_CACHE_TYPE work without flash attention? [source]
Is q8_0 KV cache safe to leave on all the time? [source]
Can I set the KV cache type per model? [source]
// KeepAlive returns the duration that models stay loaded in memory. KeepAlive can be configured via the OLLAMA_KEEP_ALIVE environment variable. // Negative values are treated as infinite. Zero is treated as no keep alive. // Default is 5 minutes. func KeepAlive() (keepAlive time.Duration) { keepAlive = 5 * time.Minute if s := Var("OLLAMA_KEEP_ALIVE"); s != "" { if d, err := time.ParseDuration(s); err == nil { keepAlive = d } else if n, err := strconv.ParseInt(s, 10, 64); err == nil { keepAlive = time.Duration(n) * time.Second } } [source]
CUDA_VISIBLE_DEVICES=GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx ollama serve [source]
[Service]Environment="OLLAMA_KV_CACHE_TYPE=q8_0" [source]
Ollama's default context length is no longer a fixed 4096. OLLAMA_CONTEXT_LENGTH compiles to 0, a sentinel meaning "decide at runtime", and the server then picks 4K, 32K, or 256K tokens from the VRAM it finds: 4096 below 23GiB, 32768 at 23GiB or more, and 262144 at 47GiB or more. [source]
OLLAMA_MAX_VRAMRemoved from production code. Only test files still reference itUse OLLAMA_GPU_OVERHEAD to reserve VRAM per GPU [source]
If you have multiple GPUs with different GFX versions, append the numeric device number to the environment variable to set them individually. For example, `HSA_OVERRIDE_GFX_VERSION_0=10.3.0` and `HSA_OVERRIDE_GFX_VERSION_1=11.0.0` [source]
Ollama’s GPU and VRAM requirements depend on model parameter count, quantization level, context window length, concurrency, and how much system RAM you have available for offloading. [source]
Ollama supports AMD GPUs through ROCm on Linux and Windows, although Windows supports fewer cards. Common supported consumer GPUs from the Radeon RX 7000 series include: [source]
Memory estimation & KV cache optimizations [source]
If you are using a container to run Ollama, make sure you've set up the container runtime first as described in [docker](./docker) [source]
Fitting a model is half the question, and speed does not track VRAM capacity. The numbers below were measured on single rented GPUs running Ollama at Q4_K_M with a 4K context, generation rate in tokens per second, one request at a time: [source]
The Ollama Docker container can be configured with GPU acceleration in Linux or Windows (with WSL2). This requires the [nvidia-container-toolkit](https://github.com/NVIDIA/nvidia-container-toolkit). See [ollama/ollama](https://hub.docker.com/r/ollama/ollama) for more details. [source]
By default models are kept in memory for 5 minutes before being unloaded. This allows for quicker response times if you're making numerous requests to the LLM. If you want to immediately unload a model from memory, use the `ollama stop` command: [source]
NVIDIA's Nova Driver Continues Leading The Way For Linux Rust GPU Drivers [source]
The stable pattern is to leave Ollama on 127.0.0.1:11434 and let nginx own the public port. TLS, credentials and request limits live in the proxy; the model runner never sees an untrusted connection directly. [source]
predictedCtx := effectiveLlamaServerContext(req.opts.NumCtx, f, numParallel) predicted := llm.PredictServerVRAM(req.model.ModelPath, f, predictedCtx) loadGpus, launchOpts = selectLlamaServerPlacement(systemInfo, gpus, predicted, req.opts) availableForBatch, _, _ := availableMemoryForPlacement(systemInfo, loadGpus, launchOpts) flashAttention := llm.LlamaServerFlashAttention(loadGpus) req.applyAutomaticGenerationBatch(completion, predictedCtx, predicted, availableForBatch, flashAttention, loadGpus) launchOpts.NumBatch = req.opts.NumBatch var batchMemory uint64 launchOpts.NumBatch, batchMemory = [source]
systemTotalMemory := systemInfo.TotalMemory systemFreeMemory := systemInfo.FreeMemory systemSwapFreeMemory := systemInfo.FreeSwap slog.Info("system memory", "total", format.HumanBytes2(systemTotalMemory), "free", format.HumanBytes2(systemFreeMemory), "free_swap", format.HumanBytes2(systemSwapFreeMemory)) [source]
oldNumCtx = req.opts.NumCtx oldNumBatch = req.opts.NumBatch effectiveNumCtx = oldNumCtx if f != nil { if trainCtx := int(f.KV().ContextLength()); trainCtx > 0 && effectiveNumCtx > trainCtx { effectiveNumCtx = trainCtx } } [source]
req.opts.NumCtx = newNumCtx predictedCtx := effectiveLlamaServerContext(req.opts.NumCtx, f, numParallel) predictedVRAM := llm.PredictServerVRAM(req.model.ModelPath, f, predictedCtx) available, _, _ := availableMemoryForPlacement(systemInfo, gpus, launchOpts) req.applyAutomaticGenerationBatch(completion, predictedCtx, predictedVRAM, available, llm.LlamaServerFlashAttention(gpus), gpus) newNumBatch = req.opts.NumBatch return oldNumCtx, effectiveNumCtx, newNumCtx, oldNumBatch, newNumBatch, true } [source]
useMmap := false req.opts.UseMMap = &useMmap req.useMMapAuto = true slog.Info("disabling mmap for llama-server load due to host memory pressure", "model", req.model.ModelPath, "model_size", format.HumanBytes2(modelSize), "loaded_mmap_size", format.HumanBytes2(loadedMmapSize), "headroom", format.HumanBytes2(mmapHostPressureHeadroom(systemInfo.TotalMemory)), "system_free", format.HumanBytes2(systemInfo.FreeMemory), "system_total", format.HumanBytes2(systemInfo.TotalMemory), "predicted_vram", format.HumanBytes2(predictedVRAM), "available_vram", format.HumanBytes2(availableVRAM), ) } [source]
pressure := modelSize + loadedMmapSize + mmapHostPressureHeadroom(systemInfo.TotalMemory) return systemInfo.FreeMemory < pressure } [source]
ollama ps should show a smaller SIZE at the same CONTEXT. The starting llama-server line should now carry --cache-type-k q8_0 --cache-type-v q8_0 and --flash-attn on. And the cache line should have changed type and size: [source]
docker run -d \ --gpus '"device=0,1"' \ -v ollama:/root/.ollama \ -p 11434:11434 \ --name ollama \ ollama/ollama [source]
所以建议把 OLLAMA_SCHED_SPREAD 当成高级调度实验项，而不是多卡必开项。先理解默认行为，再根据实际 ollama ps、日志和 nvidia-smi 观察结果调整。 [source]
Ollama FAQ：How does Ollama load models on multiple GPUs?：https://github.com/ollama/ollama/blob/main/docs/faq.mdx [source]
根据 Ollama 官方文档，使用 ollama ps、服务日志、nvidia-smi、Docker GPU 验证和 AMD 设备权限逐层定位 CPU 回退，并给出恢复与验收步骤。 [source]
Default model directories: macOS ~/.ollama/models, Linux (service install) /usr/share/ollama/.ollama/models, Windows C:\Users\%username%\.ollama\models. Setting OLLAMA_MODELS overrides it. [source]
Images consume context tokens. With Ollama's default 4096-token context, glm-ocr returned empty output; raising num_ctx to 10240 or more produced output (reported 2026-02-05). Ollama silently truncates over-long context, so the right value is hard to guess. [source]
4. Reload `systemd` and restart Ollama: [source]
How much the cache quantization impacts the model's response quality will depend on the model and the task. Models that have a high GQA count (e.g. Qwen2) may see a larger impact on precision from quantization than models with a low GQA count. [source]
You may need to experiment with different quantization types to find the best balance between memory usage and quality. [source]
Published tokens per second figures come from someone else's GPU, model, quantisation, context length and prompt. None of those match yours, so treat any number you read as a rough hint and measure the machine in front of you. [source]
If you want the speed side of the trade, run the same prompt with ollama run llama3.1:8b --verbose before and after and compare the eval rate it prints. Flash attention changes the attention kernel, and quantized cache values must be dequantized on every read, so the tokens-per-second figure can move in either direction on a CPU box. Measuring tokens per second properly is a few minutes of work and is the only honest answer for your hardware. [source]
Here’s how context length affects total VRAM for Llama 3.1 8B Q4_K_M: [source]
This can be overridden with the `OLLAMA_CONTEXT_LENGTH` environment variable. For example, to set the default context window to 8K, use: [source]
Ollama binds 127.0.0.1 port 11434 by default. Change the bind address with the `OLLAMA_HOST` environment variable. [source]
• Q4_0 - Some noticeable quality reduction, but still usable for those without much vRAM or working on creative tasks where quality is less critical. In early testing, Q4_0 added around 0.206-0.25~ perplexity to the model. [source]
* **AMD GPU**: `ollama-windows-amd64-rocm.zip` * **MLX (CUDA)**: `ollama-windows-amd64-mlx.zip` [source]
Alternatively, create an override file manually in `/etc/systemd/system/ollama.service.d/override.conf`: [source]
RX 7900 XTX on Vulkan: typical runs on the RADV driver sit near 158 tok/s, an efficiency of 0.73. The best proprietary-driver runs on the scoreboard reach about 183, so those users can expect about 15 to 20 percent more. [source]
systemd as the init system. The official install script registers ollama.service. [source]
OLLAMA_KEEP_ALIVE How long a model stays in VRAM 10m [source]
Dedicated NVIDIA GPUs from German data centres, billed by the hour and ready in minutes. Rent a GPU server → [source]
403 Forbidden only through the proxy. Ollama rejected the forwarded origin. Add proxy_set_header Host 127.0.0.1:11434; to the location block, or put the external hostname into OLLAMA_ORIGINS and restart the service. [source]
if m.CheckCapabilities(model.CapabilityVision) == nil { // multimodal models require at least 2048 context opts.NumCtx = max(opts.NumCtx, 2048) } [source]
s.loadedMu.Lock() if oldRunner, ok := s.loaded[runner.modelKey]; ok { // Shouldn't happen, but safeguard against leaking a runner slog.Warn("model was still loaded", "old_runner", oldRunner, "new_runner", runner) oldRunner.refMu.Lock() oldRunner.unload() oldRunner.refMu.Unlock() } s.activeLoading = nil s.loaded[runner.modelKey] = runner slog.Info("loaded runners", "count", len(s.loaded)) s.loadedMu.Unlock() [source]
func hasCUDADevice(gpus []ml.DeviceInfo) bool { return slices.ContainsFunc(gpus, func(gpu ml.DeviceInfo) bool { return gpu.Library == "CUDA" }) } [source]
func availableMemoryForLoad(systemInfo ml.SystemInfo, gpus []ml.DeviceInfo) (available, gpuFree uint64, systemLimited bool) { var sharedGPUFree uint64 var discreteGPUFree uint64 for _, gpu := range gpus { gpuFree += gpu.FreeMemory if gpu.Integrated { sharedGPUFree += gpu.FreeMemory } else { discreteGPUFree += gpu.FreeMemory } } [source]
func availableMemoryForGPU(systemInfo ml.SystemInfo, gpu ml.DeviceInfo) uint64 { if gpu.Integrated && systemInfo.FreeMemory > 0 && systemInfo.FreeMemory < gpu.FreeMemory { return systemInfo.FreeMemory } [source]
func (s *Scheduler) applyLlamaServerMmapDefaults(req *LlmRequest, launchOpts api.Options, systemInfo ml.SystemInfo, gpus []ml.DeviceInfo, f *gguf.Model, numParallel int) api.Options { predictedCtx := effectiveLlamaServerContext(req.opts.NumCtx, f, numParallel) predictedVRAM := llm.PredictServerVRAM(req.model.ModelPath, f, predictedCtx) availableVRAM, _, _ := availableMemoryForPlacement(systemInfo, gpus, launchOpts) [source]
func disableMmapDefaultReason(goos string, opts api.Options, gpus []ml.DeviceInfo, blockCount, predictedVRAM, availableVRAM uint64) string { if opts.UseMMap != nil { return "" } if opts.NumGPU == 0 || len(gpus) == 0 || allDevicesLibrary(gpus, "cpu") { return "cpu" } if goos == "windows" && hasDeviceLibrary(gpus, "cuda") { return "windows_cuda" } if hasDeviceLibrary(gpus, "metal") { if opts.NumGPU > 0 && blockCount > 0 && uint64(opts.NumGPU) < blockCount+1 { return "metal_partial_offload" } if opts.NumGPU < 0 && predictedVRAM > 0 && availableVRAM > 0 && predictedVRAM > availableVRAM { return "m [source]
func (s *Scheduler) maybeDisableMmapForHostPressure(req *LlmRequest, launchOpts api.Options, systemInfo ml.SystemInfo, gpus []ml.DeviceInfo, f *gguf.Model, numParallel int) { modelSize := modelFileSize(req.model.modelPaths()...) loadedMmapSize := s.loadedMmapModelSizeLocked() predictedCtx := effectiveLlamaServerContext(req.opts.NumCtx, f, numParallel) predictedVRAM := llm.PredictServerVRAM(req.model.ModelPath, f, predictedCtx) availableVRAM, _, _ := availableMemoryForPlacement(systemInfo, gpus, launchOpts) placementGpus := gpusForPlacement(gpus, launchOpts) [source]
func (s *Scheduler) updateFreeSpace(allGpus []ml.DeviceInfo) { if len(allGpus) == 0 { return } predMap := map[ml.DeviceID]uint64{} // Sum up the total predicted usage per GPU for all runners s.loadedMu.Lock() runners := make([]*runnerRef, 0, len(s.loaded)) for _, r := range s.loaded { runners = append(runners, r) } s.loadedMu.Unlock() for _, r := range runners { r.refMu.Lock() if r.llama != nil { for _, gpu := range allGpus { predMap[gpu.DeviceID] += r.llama.VRAMByGPU(gpu.DeviceID) } } else { slog.Warn("unexpected nil runner reference, memory prediction may be incorrect") } r.refMu.Unlock() } [source]
// In the future we can enhance the algorithm to be smarter about picking the optimal runner to unload // e.g., if we have multiple options, will one make room for the request? sort.Sort(ByDurationAndName(runnerList)) [source]
// First try to find a runner that's already idle for _, runner := range runnerList { runner.refMu.Lock() rc := runner.refCount runner.refMu.Unlock() if rc == 0 { slog.Debug("found an idle runner to unload", "runner", runner) return runner } } // None appear idle, just wait for the one with the shortest duration slog.Debug("no idle runners, picking the shortest duration", "runner_count", len(runnerList), "runner", runnerList[0]) return runnerList[0] } [source]
// refMu must not be acquired while holding loadedMu: the expiration path // locks them in the opposite order. models := make([]loadedModel, 0, len(runners)) for _, r := range runners { r.refMu.Lock() if r.model == nil { // Unloaded after the snapshot above was taken r.refMu.Unlock() continue } lm := loadedModel{ model: r.model, runner: r.runner, size: int64(r.totalSize), sizeVRAM: int64(r.vramSize), expiresAt: r.expiresAt, } if r.llama != nil { lm.contextLength = r.llama.ContextLength() total, vram := r.llama.MemorySize() lm.size = int64(total) lm.sizeVRAM = int64(vram) } // The [source]
Why doubling num_ctx cost more than the weight quantization saved [source]
Unsupported cache type: q8. The value is passed to the runner as a string and matched exactly. Ollama lowercases it for you, so Q8_0 works, but q8 and int8 do not. Use f16, q8_0 or q4_0, which are the three the Ollama documentation lists. [source]
Nothing changed, and the journal never shows the variable. You set it with export in your shell before running ollama run. That reaches the CLI, which is a client, and never reaches the server, which is the systemd unit with its own environment. systemctl show ollama -p Environment is the only view that counts. [source]
This was confirmed in #20866 and by our testing with Gemma 4 12B QAT + MTP (head_size=512, SWA, Blackwell RTX 5070 Ti). [source]
I am running it on Kubuntu 26.04 with Vulkan backend. [source]
// Host returns the scheme and host. Host can be configured via the OLLAMA_HOST environment variable. // Default is scheme "http" and host "127.0.0.1:11434" func Host() *url.URL { defaultPort := "11434" [source]
images: (optional) a list of base64-encoded images (for multimodal models such as llava) [source]
images (optional): a list of images to include in the message (for multimodal models such as llava) [source]
Multi-GPU performance: Ollama will now schedule models more efficiently over multiple GPUs, significantly improving multi-GPU and mismatched GPU performance [source]
Ollama ROCm Docker container crashing on RX 6650 #2870 [source]
Ollama ROCm Docker container crashing on RX 6650#2870 [source]
AMD 多卡比 NVIDIA 更容易遇到驱动、ROCm 版本、GFX 版本支持的问题。官方文档中也提到 Linux 下 ROCm 驱动版本、HSA_OVERRIDE_GFX_VERSION 等兼容性处理。多张不同代 AMD 卡混用时，先确认每张卡是否单独可用，再考虑多卡。 [source]
在一些多 GPU 配置讨论里，会看到 OLLAMA_SCHED_SPREAD=1 或 OLLAMA_SCHED_SPREAD=true。它和 Ollama 的调度策略有关，常被用于希望模型或请求更分散地利用多张 GPU 的场景。 [source]
macOS app: the menu-bar app is launched by launchd, not by your shell, so it never sees ~/.zshrc. Ollama's FAQ specifies launchctl for this case. Set the variable, then quit and reopen Ollama. [source]
Can Ollama offload the KV cache to system RAM? [source]
OLLAMA_VULKANEnable the Vulkan backendon where the backend is installedWindows and Linux [source]
GGML_VK_VISIBLE_DEVICESWhich Vulkan devices are visible, by numeric IDallWindows and Linux [source]
Flash attentionAutomatic where the backend and devices support itAutomatic, resolves on for Metal [source]
> **NOTE:** > Additional AMD GPU support is provided by the Vulkan Library - see below. [source]
Ollama requires an AMD ROCm v7 / HIP7-capable driver stack on Windows. [source]
Ollama supports GPU acceleration on Apple devices via the Metal API. [source]
According to Ollama’s docs, you can set the context length with the OLLAMA_CONTEXT_LENGTH environment variable. [source]
Open `Control Panel > Networking and Internet > View network status and tasks` and click on `Change adapter settings` on the left panel. Find the `vEthernet (WSL)` adapter, right click and select `Properties`. Click on `Configure` and open the `Advanced` tab. Search through each of the properties until you find `Large Send Offload Version 2 (IPv4)` and `Large Send Offload Version 2 (IPv6)`. *Disable* both of these properties. [source]
Third-party guides say newer macOS and Windows desktop apps have a Settings toggle 'Expose Ollama to the network'. This came only from search-result snippets; the guide page returned HTTP 403 and no official page confirms it. [source]
If the GPU host should stay off the public network entirely, terminate TLS on a separate gateway. A small scalable Linux VM is enough for nginx, and the GPU machine then only accepts connections from that one private address. [source]
func InitScheduler(ctx context.Context) *Scheduler { maxQueue := envconfig.MaxQueue() sched := &Scheduler{ pendingReqCh: make(chan *LlmRequest, maxQueue), finishedReqCh: make(chan *LlmRequest, maxQueue), expiredCh: make(chan *runnerRef, maxQueue), unloadedCh: make(chan any, maxQueue), loaded: make(map[string]*runnerRef), newServerFn: llm.NewLlamaServer, getGpuFn: discover.GPUDevices, getSystemInfoFn: discover.GetSystemInfo, waitForRecovery: 5 * time.Second, } sched.loadFn = sched.load return sched } [source]
func (s *Scheduler) processPending(ctx context.Context) { maxRunners := envconfig.MaxRunners() [source]
if !envconfig.SchedSpread() && predictedVRAM > 0 { gpu, available, ok := bestSingleGPUFit(systemInfo, groups, predictedVRAM) if ok { selected, launchOpts := singleLlamaServerGPUPlacement(gpu, launchOpts) slog.Info("selecting single GPU for llama-server model", "main_gpu", *launchOpts.MainGPU, "id", gpu.ID, "filter_id", gpu.FilterID, "library", gpu.Library, "name", gpu.Name, "description", gpu.Description, "integrated", gpu.Integrated, "predicted", format.HumanBytes2(predictedVRAM), "available", format.HumanBytes2(available)) logSelectedGPUGroup(gpus, selected) return selected, launchOpts } } [source]
For best performance, use the maximum context length for a model, and avoid offloading the model to CPU. Verify the split under `PROCESSOR` using `ollama ps`. [source]
docker run -d \ --gpus=all \ -v ollama:/root/.ollama \ -p 11434:11434 \ --name ollama \ ollama/ollama [source]
Ollama GPU 文档：Hardware support / GPU Selection：https://github.com/ollama/ollama/blob/main/docs/gpu.mdx [source]
Ollama Docker Hub：https://hub.docker.com/r/ollama/ollama [source]
Verify GPU acceleration with ollama ps to check where the model is loaded and nvidia-smi to confirm that the GPU is available to Ollama. [source]
I've built an interactive VRAM estimator to help you understand the impact of K/V context cache quantisation on your VRAM usage. You can adjust the model size, context size and quantisation level to see how it affects the memory requirements. [source]
Llama 3.1 8B Q4_K_M, RTX 4090147 tok/s148 tok/sOllama run [source]
Llama 3.1 8B Q4_K_M, RTX 3090138 tok/s119 tok/sOllama run (worst fit) [source]
Qwen2.5 32B Q4_K_M, RTX 509070 tok/s71 tok/sOllama run [source]
Weights are parameters multiplied by bytes per parameter. Dividing the real Llama 3.3 70B GGUF files by its 70.6 billion parameters gives the effective rate for each quantization: [source]
How fast will it run? Measured tokens per second per GPU [source]
You want to runMinimum VRAM (Q4_K_M, 32K context)Practical card [source]
At Q4_K_M quantization, model file sizes scale roughly like this: [source]
CUDA_VISIBLE_DEVICESWhich NVIDIA devices are visible. UUIDs recommended over numeric IDsallWindows and Linux [source]
General Multimodal Understanding & Reasoning [source]
Official docs document the desktop app's Settings only for context length (launch post, July 2025) and sign-in to Ollama cloud (Settings page on Mac and Windows). Neither the launch post nor the macOS and Windows docs pages mention an app setting for network exposure or model location. [source]
Ollama server can be configured with environment variables. [source]
If a different directory needs to be used, set the environment variable `OLLAMA_MODELS` to the chosen directory. [source]
ollama 0.12.11 Brings Vulkan Acceleration [source]
ollama 0.12.11 released this week as the newest feature update to this easy-to-run method of deploying OpenAI GPT-OSS, DeepSeek-R1, Gemma 3, and other large language models. Exciting with ollama 0.12.11 is that it's now supporting the Vulkan API. [source]
This commit in ollama 0.12.11 lays out all the details on the Vulkan API support for ollama. Over the past few weeks ollama offered the Vulkan support as experimental. [source]
Start Ollama and Q8_0 quantisation will be used for the K/V context cache by default. [source]
If you run into issues with quality however - you can simply disable K/V context cache quantisation by setting OLLAMA_KV_CACHE_TYPE environment variable to f16 or not setting it at all. This is even easier if you run Ollama in a container as you can simply run two containers with different configurations (as there's practically no overhead to running multiple containers). [source]
The most commonly used quantisation levels for the K/V are Q8_0 and Q4_0, unquantised is referred to as F16 (or F32 although for inference you would not run F32). [source]
• Q8_0 - Minimal quality impact for normal text generation models, suitable for most users to be enabled by default. Perplexity measurements on an early implementation showed it added around 0.002-0.05~ perplexity to the model. [source]
I recently performed some perplexity measurements comparing Q8_0 and Q4_0 quantisation levels on a Qwen 2.5 Coder 7B model as the Qwen series of models is known to be quite sensitive to quantisation. [source]
Note that the ability to set the K/V cache quantisation level in a model's Modelfile was removed from the PR, but I hope that Ollama will reconsider this in the future so people can experiment with different quantisation levels for different models and use cases. [source]
K/V context cache quantisation requires Flash Attention to be enabled. Enabling Flash Attention has no negative impacts and is something I expect to become the default behaviour in the near future with Ollama. [source]
• Apple Silicon (Metal): Works on Apple Silicon devices. [source]
• HuggingFace's blog post on K/V cache quantisation, which provides a more technical deep dive into the topic in the context of Transformers [source]
3. Edit or create a new variable for your user account for `OLLAMA_MODELS` where you want the models stored [source]
The easiest way to install Ollama on Windows is to use the `OllamaSetup.exe` installer. It installs in your account without requiring Administrator rights. We update Ollama regularly to support the latest models, and this installer will help you keep up to date. [source]
This allows for embedding Ollama in existing applications, or running it as a system service via `ollama serve` with tools such as [NSSM](https://nssm.cc/). [source]
If you are using OLLAMA_MODELS environment variable to set the models path, then you can change the models path to a directory that you have permission to read. For example, you can create a directory in your home directory and set the models path to that directory. And then in Msty, update the models path to the new directory by following the steps in Use existing Ollama Models. [source]
Troubleshooting common issues with AMD ROCm on Windows [source]
Qwen3.8 27B KV cache per token (Ollama log, 8,192 cells)64 KiB512 MiB / 8,192 = 64 KiB [source]
Llama 2 7B Q4_0, Apple M5 Max118.5 tok/s119.9 tok/sllama.cpp Apple scoreboard [source]
KV cache per token for popular 2026 models [source]
The formula: weights plus KV cache plus overhead [source]
The practical rule that falls out of all three: pick a quantization that leaves at least one gigabyte of headroom below the VRAM of the card, never one that exactly equals it. [source]
For processing large documents, Ollama’s context length can be increased in the settings. Note: this will require more memory. [source]
The queue size is controlled by OLLAMA_MAX_QUEUE (default: 512 requests). [source]
For GPU inference, requires full VRAM allocation per model - partial loads aren’t supported. [source]
OLLAMA_MAX_QUEUE Maximum queued requests 512 [source]
Only qwen3:30b-a3b showed the same CPU/GPU spread, three other models were pushed more to CPU in new version. In my tests, to my disappointment, new version of Ollama is worse, and these results are contradicting post on Ollama blog. [source]
Model Old ver: VRAM allocated Old ver: CPU/GPU New ver: VRAM allocated New ver: CPU/GPU [source]
IoT in the cloudConnect devices and sensor data Predictive maintenance & analyticsPut machine data to use ERP hostingRun your ERP reliably CAD in the cloudDesign with GPU performance CFD simulationFlow simulation on demand Manufacturing web hostingPublish product data securely [source]
OLLAMA_ORIGINS Browser origins allowed to call the API https://chat.example.com [source]
OLLAMA_MODELS Directory holding model blobs /srv/ollama/models [source]
OLLAMA_MAX_LOADED_MODELS Models resident at the same time 2 [source]
func effectiveModelContext(numCtx int, f *gguf.Model) int { return effectiveContext(numCtx, modelTrainContext(f)) } [source]
func effectiveContext(numCtx, trainCtx int) int { if trainCtx > 0 && numCtx > trainCtx { return trainCtx } [source]
// Update free memory from currently loaded models logutil.Trace("updating free space", "gpu_count", len(gpus), "model", pending.model.ModelPath) s.updateFreeSpace(gpus) [source]
func effectiveLlamaServerContext(numCtx int, f *gguf.Model, numParallel int) int { return effectiveModelContext(numCtx, f) * max(numParallel, 1) } [source]
func nextLowerAutoNumCtx(numCtx int) (int, bool) { switch { case numCtx > 32768: return 32768, true case numCtx > 4096: return 4096, true default: return 0, false } } [source]
func availableMemoryForPlacement(systemInfo ml.SystemInfo, gpus []ml.DeviceInfo, opts api.Options) (available, gpuFree uint64, systemLimited bool) { placementGpus := gpusForPlacement(gpus, opts) if len(placementGpus) == 1 && opts.MainGPU != nil { gpuFree = placementGpus[0].FreeMemory available = availableMemoryForGPU(systemInfo, placementGpus[0]) systemLimited = available < gpuFree return available, gpuFree, systemLimited } [source]
func disableMmapForHostPressure(goos string, opts api.Options, systemInfo ml.SystemInfo, gpus []ml.DeviceInfo, modelSize, loadedMmapSize, predictedVRAM, availableVRAM uint64) bool { if opts.UseMMap != nil || goos != "linux" || modelSize == 0 || systemInfo.FreeMemory == 0 || !allDiscreteGPUs(gpus) { return false } [source]
ctx, cancel := context.WithTimeout(ctx, timeout) defer cancel() if !reflect.DeepEqual(runner.model.AdapterPaths, req.model.AdapterPaths) || // have the adapters changed? !reflect.DeepEqual(runner.model.ProjectorPaths, req.model.ProjectorPaths) || // have the projectors changed? (!runner.model.IsMLX() && !reflect.DeepEqual(optsExisting, optsNew)) || // have the runner options changed? runner.llama.Ping(ctx) != nil { return true } [source]
func (s *Scheduler) expireRunner(model *Model) { modelKey := schedulerModelKey(model) var runners []*runnerRef [source]
OLLAMA_NUM_PARALLEL is how many requests one loaded model handles at the same time. The default is 1, so requests are served one after another. [source]
Copy sudo systemctl daemon-reload sudo systemctl restart ollama systemctl show ollama --property=Environment [source]
Ollama logs its entire environment at startup on a line whose message is server config. That map shows exactly what the server read, and it is the fastest way to settle an argument about whether a variable took effect. It shows the value you set, not the number of slots a particular model actually got. For that, read the -np value on the runner line above. [source]
Why a smaller MAX_QUEUE is often the better setting [source]
Does raising OLLAMA_NUM_PARALLEL make Ollama faster? [source]
How many parallel slots should I set? [source]
ChartKV cache for llama3.1:8b by num_ctx, computed, GiB [source]
Chartllama3.1:8b: weight file size against f16 KV cache, GiB [source]
Copy sudo journalctl -u ollama --since "5 min ago" --no-pager | grep -E 'starting llama-server|llama_kv_cache: size|Flash Attention' [source]
The last command should print one Environment= line holding the installer's PATH= plus your two variables. If they are missing, the drop-in did not take, and the section on failures below covers why. Restarting the service unloads every model, so this is a change to make between sessions, not during one. [source]
Repeat the baseline exactly: the same model and the same num_ctx, with the same commands. [source]
Ollama sets the key cache and the value cache to the same type; there is no way to quantize one and not the other through this setting. So the practical rule is: q8_0 is the setting you leave on. q4_0 is the setting you test for your task before you trust it. A cheap test is a long document with one specific fact planted in the middle, a question that needs that fact, and the same request at f16 and q4_0 with "temperature": 0 in the options. If the answers diverge, that is your task's verdict, and no general benchmark overrides it. Set the cache type back with the same two commands, daemon-rel [source]
Copy quantized V cache was requested, but this requires Flash Attention [source]
The Ollama FAQ calls its precision loss "very small", and its memory is about 53 percent of f16 by the block layout, so for most servers it is the right default. The setting is global, so every model the server loads gets it. Test it once on your own task with "temperature": 0 at f16 and at q8_0, and if the outputs match you are done. [source]
Not with this setting. The documentation states it is a global option that applies to every model the server runs. The controls you do have per model are num_ctx, which is what the cache size scales with, and the weight file you pull. If one model needs an f16 cache and another does not, the honest answer is a second Ollama instance on a different port with its own drop-in. [source]
Why did raising num_ctx use more memory than the weights did? [source]
Setting a larger context length will increase the amount of memory required to run a model. Ensure you have enough VRAM available to increase the context length. [source]
Change the slider in the Ollama app under settings to your desired context length. [source]
s := strings.TrimSpace(Var("OLLAMA_HOST")) scheme, hostport, ok := strings.Cut(s, "://") switch { case !ok: scheme, hostport = "http", s if s == "ollama.com" { scheme, hostport = "https", "ollama.com:443" } case scheme == "http": defaultPort = "80" case scheme == "https": defaultPort = "443" } [source]
// ConnectableHost returns Host() with unspecified bind addresses (0.0.0.0, ::) // replaced by the corresponding loopback address (127.0.0.1, ::1). // Unspecified addresses are valid for binding a server socket but not for // connecting as a client, which fails on Windows. func ConnectableHost() *url.URL { u := Host() host, port, err := net.SplitHostPort(u.Host) if err != nil { return u } [source]
// AllowedOrigins returns a list of allowed origins. AllowedOrigins can be configured via the OLLAMA_ORIGINS environment variable. func AllowedOrigins() (origins []string) { if s := Var("OLLAMA_ORIGINS"); s != "" { origins = strings.Split(s, ",") } [source]
// Models returns the path to the models directory. Models directory can be configured via the OLLAMA_MODELS environment variable. // Default is $HOME/.ollama/models func Models() string { if s := Var("OLLAMA_MODELS"); s != "" { return s } [source]
// LogLevel returns the log level for the application. // Values are 0 or false INFO (Default), 1 or true DEBUG, 2 TRACE func LogLevel() slog.Level { level := slog.LevelInfo if s := Var("OLLAMA_DEBUG"); s != "" { if b, _ := strconv.ParseBool(s); b { level = slog.LevelDebug } else if i, _ := strconv.ParseInt(s, 10, 64); i != 0 { level = slog.Level(i * -4) } } [source]
var ( LLMLibrary = String("OLLAMA_LLM_LIBRARY") Editor = String("OLLAMA_EDITOR") [source]
keep_alive: controls how long the model will stay loaded into memory following the request (default: 5m) [source]
If an empty prompt is provided and the keep_alive parameter is set to 0, a model will be unloaded from memory. [source]
If the messages array is empty and the keep_alive parameter is set to 0, a model will be unloaded from memory. [source]
The files parameter should include the file name and SHA256 digest of each file in the Safetensors model. Use /api/blobs/:digest to push each file before calling this API. Files remain in the cache until the Ollama server is restarted. [source]
Ensures that the file blob (Binary Large Object) used with create a model exists on the server. This checks your Ollama server and not ollama.com. [source]
Push a file to the Ollama server to create a "blob" (Binary Large Object). [source]
The bundled Ollama plugin registers Ollama as an image-capable media-understanding provider, so OpenClaw can route explicit image-description requests and configured image-model defaults through local or hosted Ollama vision models. [source]
If you define `models.providers.ollama.models` manually, mark vision models explicitly: [source]
Vision models accept images alongside text so the model can describe, classify, and answer questions about what it sees. [source]
Ollama FAQ 里对多 GPU 加载逻辑说得比较直接：加载新模型时，Ollama 会估算模型所需 VRAM，并和当前可用显存比较。如果模型能完整放进某一张 GPU，它会加载到那张 GPU 上；如果单张 GPU 放不下，才会分布到所有可用 GPU 上。 [source]
NVIDIA 多卡环境下，用 CUDA_VISIBLE_DEVICES 控制 Ollama 能看到哪些卡。 [source]
GPU 0: NVIDIA GeForce RTX 3090 (UUID: GPU-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx) GPU 1: NVIDIA GeForce RTX 3070 (UUID: GPU-yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy) [source]
[Service] Environment="CUDA_VISIBLE_DEVICES=0,1" [source]
如果容器里 nvidia-smi 看不到卡，Ollama 也不可能用到 GPU。先排查 Docker GPU passthrough，再排查 Ollama。 [source]
[Service] Environment="OLLAMA_SCHED_SPREAD=true" [source]
不一定。Linux systemd 服务、Windows 后台应用、Docker 容器都可能需要重启，才能重新识别设备和环境变量。 [source]
Wan 2.2 本地部署指南，说明模型家族、ComfyUI 与 Diffusers 路线、CUDA 环境、显存预算、模型下载、生成参数、性能验证和常见错误。 [source]
Is Ollama flash attention on by default? [source]
OLLAMA_ORIGINSAllowed CORS origins for browser requestslocalhost, 127.0.0.1 and 0.0.0.0 on any port over http and https, plus app://*, file://*, tauri://*, vscode-webview://*, vscode-file://** [source]
OLLAMA_FLASH_ATTENTIONThree-state flash attention overrideunset = automatic where supported1 to force on, 0 to force off [source]
OLLAMA_CONTEXT_LENGTHDefault context window size0, meaning auto: 4K under 23GiB VRAM, 32K at 23GiB, 256K at 47GiB8192 [source]
Ollama Environment Variables: GPU Selection and Backends [source]
Ollama reads environment variables once, at server startup. Quit the app fully and reopen it, or stop and restart ollama serve. Reloading a client or browser tab is not enough. [source]
Frequently Asked Questions Is flash attention enabled by default in Ollama? [source]
If manually running `ollama serve` in a terminal, the logs will be on that terminal. [source]
You can set OLLAMA\_LLM\_LIBRARY to any of the available LLM libraries to bypass autodetection, so for example, if you have a CUDA card, but want to force the CPU LLM library with AVX2 vector support, use: [source]
In some Linux distributions, SELinux can prevent containers from accessing the AMD GPU devices. On the host system you can run `sudo setsebool container_use_devices=1` to allow containers to use devices. [source]
OLLAMA_CONTEXT_LENGTH environment variable didn’t have an effect, but there’s another way [source]
What determines Ollama GPU and VRAM requirements? [source]
Ollama GPU requirements: VRAM, RAM, and supported GPUs [source]
The main factors that determine where you land in that range include model parameter count, quantization level, and context window length. [source]
What more memory gives youMore RAM lets you run larger models on the CPUMore VRAM lets you run larger models and longer contexts on the GPU [source]
The more parameters a model has, the more VRAM you need to run it on the GPU. [source]
System RAM gives Ollama a place to store model data that doesn’t fit in VRAM, while storage determines how many models and quantization versions you can keep downloaded. [source]
Start with CPU-only mode. Run the following command with num_gpu set to 0: [source]
Thank you to the teams working to release openly available and powerful vision models - Google DeepMind, Meta Llama, Alibaba Qwen, Mistral, IBM Granite, and the many countless people & labs in the community. [source]
Ollama runs an HTTP server and can be exposed using a proxy server such as Nginx. To do so, configure the proxy to forward requests and optionally set required headers (if not exposing Ollama on the network). For example, with Nginx: [source]
$ sudo apt install -y apache2-utils $ sudo htpasswd -c /etc/nginx/ollama.htpasswd ollama-client [source]
sessionDuration := envconfig.KeepAlive() if req.sessionDuration != nil { sessionDuration = req.sessionDuration.Duration } [source]
req.opts.NumBatch = automaticGenerationBatch(effectiveCtx, predictedVRAM, availableMemory, flashAttention, gpus) } [source]
func mmapHostPressureHeadroom(totalMemory uint64) uint64 { if totalMemory == 0 { return 8 * format.GigaByte } return max(8*format.GigaByte, totalMemory/10) } [source]
Accurate reporting: Measurements in tools like nvidia-smi will now match ollama ps making it easy to track memory utilization on your system [source]
This reference is audited against Ollama v0.32.5, released 2026-07-27. Every default below was read from envconfig/config.go at that tag. Nothing in that file changed between v0.30.10 (2026-06-17) and v0.32.5: the file is byte-identical across both, so no variable was added, removed, or had its default changed in that window. [source]
ollama stop llama3.1:8b-instruct-q4_K_M [source]
Flash AttentionA technique used to reduce the memory requirements of LLMs by only attending to a subset of the context at a time. [source]
Llama 3.3 70B Q4_K_M, L40S16.4 tok/s16 tok/sOllama run [source]
Qwen3.5 9B Q4_K_M, A10G80 tok/s82 tok/sllama-bench [source]
ModelWeights (Q4_K_M)KV per tokenKV at 32KKV at 128KTotal at 32K [source]
VRAMCardApprox. price (30 Sep 2026)Runs at Q4_K_M [source]

Related concepts

GPU — is a related of Ollama server operations
VRAM — is a synonym of Ollama server operations
kv cache — is a synonym of Ollama server operations
cache type — is a synonym of Ollama server operations
num_parallel — is a synonym of Ollama server operations
runner — is a part of Ollama server operations
flash attention — is a synonym of Ollama server operations
ROCm — is a synonym of Ollama server operations
parallel — is a related of Ollama server operations
max_loaded_models — is a synonym of Ollama server operations
Vulkan — is a synonym of Ollama server operations
device — is a related of Ollama server operations
CUDA — is a synonym of Ollama server operations
quantization — is a measure of Ollama server operations
Q4_K_M — is a measure of Ollama server operations
num_ctx — is a synonym of Ollama server operations
model eviction — is a synonym of Ollama server operations
context length — is a synonym of Ollama server operations
backend — is a related of Ollama server operations
systemd — is a synonym of Ollama server operations

Children

Apple Silicon unified memory limits for Ollama (frontier)
KV cache sizing per token by architecture (frontier)
MoE model memory sizing (gpt-oss, DeepSeek) (frontier)
OCR models on Ollama (frontier)
OLLAMA_GPU_OVERHEAD tuning (frontier)
Ollama /api/ps and ollama stop (frontier)
Ollama GPU backends and hardware support: CUDA / ROCm / Vulkan / device selection env vars
Ollama OLLAMA_ORIGINS and CORS (frontier)
Ollama ROCm override for unsupported GFX targets (frontier)
Ollama behind a reverse proxy (frontier)
Ollama desktop app: settings / network exposure / model dir
Ollama flash attention and KV cache type settings
Ollama hardware sizing: RAM and VRAM requirements by model and quant
Ollama in Docker and Kubernetes
Ollama in Docker with GPU passthrough (frontier)
Ollama installation and service ops on Linux and Windows
Ollama integrated GPU handling (frontier)
Ollama load timeout (OLLAMA_LOAD_TIMEOUT) (frontier)
Ollama memory estimation and multi-GPU placement: SCHED_SPREAD / layer offload fit
Ollama multimodal embeddings (frontier)
Ollama num_gpu layer count option (frontier)
Ollama reverse proxy and authentication (frontier)
Ollama server scheduling: NUM_PARALLEL / MAX_LOADED_MODELS / MAX_QUEUE / keep_alive eviction
Ollama vision and multimodal usage (frontier)
Ollama vision context sizing (frontier)
llama.cpp --fit auto-fit (frontier)
llama.cpp flash attention kernels and CUDA build flags (frontier)

Frontier under this node: Apple Silicon unified memory limits for Ollama, KV cache sizing per token by architecture, MoE model memory sizing (gpt-oss, DeepSeek), OCR models on Ollama, OLLAMA_GPU_OVERHEAD tuning, Ollama /api/ps and ollama stop, Ollama OLLAMA_ORIGINS and CORS, Ollama ROCm override for unsupported GFX targets, Ollama behind a reverse proxy, Ollama in Docker with GPU passthrough, Ollama integrated GPU handling, Ollama load timeout (OLLAMA_LOAD_TIMEOUT), Ollama multimodal embeddings, Ollama num_gpu layer count option, Ollama reverse proxy and authentication, Ollama vision and multimodal usage, Ollama vision context sizing, llama.cpp --fit auto-fit, llama.cpp flash attention kernels and CUDA build flags

← the whole tree · 3D view· how to read this page