Scaling Ollama across replicas: load balancing and when to move to vLLM

Parent: Ollama templates and scale-out · Published reference · snapshot 2026-10-14 · skill ollama-expert/references/ollama-templates-and-scale-out.md

Also known as: Ollama scale-out and vLLM

↓ Facts as markdown all context files

Everything the scanned sources say about Ollama scale-out and vLLM: 53 source-anchored units across 6 sources, grouped by facet.

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Definitions

Hybrid load balancing sits between the internal and external approaches. Each node runs its own API server(s) that only queue requests to the data-parallel engines colocated on that node. An upstream load balancer (for example, an ingress controller or traffic router) spreads user requests across those per-node endpoints. [source]
vLLM supports Data Parallel deployment, where model weights are replicated across separate instances/GPUs to process independent batches of requests. [source]
vLLM supports "self-contained" data parallel deployments that expose a single API endpoint. [source]

Structure and components

Requests for one model name are routed across multiple deployments that share the same model_name in model_list. Each deployment entry has its own litellm_params: the model, api_base, api_key and an optional rpm rate limit for that deployment. [source]
In these cases, the data parallel ranks are not completely independent. Forward passes must be aligned, and expert layers across all ranks are required to synchronize during every forward pass, even when there are fewer requests to be processed than DP ranks. [source]
There are two distinct modes supported for online deployments - self-contained with internal load balancing, or externally per-rank process deployment and load balancing. [source]
For MoE models, particularly those like DeepSeek that employ MLA (Multi-head Latent Attention), it can be advantageous to use data parallel for the attention layers and expert or tensor parallel (EP or TP) for the expert layers. [source]

How it works

In all cases, it is beneficial to load-balance requests between DP ranks. For online deployments, this balancing can be optimized by taking into account the state of each DP engine - in particular its currently scheduled and waiting (queued) requests, and KV cache state. Each DP engine has an independent KV cache, and the benefit of prefix caching can be maximized by directing prompts intelligently. [source]
In vLLM, each DP rank is deployed as a separate "core engine" process that communicates with front-end process(es) via ZMQ sockets. Data Parallel attention can be combined with Tensor Parallel attention, in which case each DP engine owns a number of per-GPU worker processes equal to the configured TP size. [source]
When using --data-parallel-external-lb with --nnodes, the data parallel rank can be inferred from --node-rank only when --nnodes is evenly divisible by --data-parallel-size. If it is not evenly divisible, automatic rank inference is rejected and --data-parallel-rank must be set explicitly. When multiple external-LB ranks are co-located on a node, assign each process a disjoint device set with CUDA_VISIBLE_DEVICES or --device-ids; devices are not partitioned automatically between processes. [source]
By contrast, the admission control limits --max-num-queued-reqs and --max-num-queued-tokens apply to the whole server, not per DP rank: each API server process counts in-flight requests and prefill backlog across all DP ranks it routes to. For example, with --data-parallel-size=4 --max-num-seqs=256 --max-num-queued-reqs=256, the deployment rejects new requests once 256 are in-flight in total, even though ranks could jointly run 1024. To keep ranks saturating, size the cap as roughly data-parallel-size * max-num-seqs plus the desired queue depth. [source]
Currently, the internal DP load balancing is done within the API server process(es) and is based on the running and waiting queues in each of the engines. This could be made more sophisticated in future by incorporating KV cache aware logic. [source]
Weighted failover is only active when routing_strategy is simple-shuffle (the default). It is capped by max_fallbacks (default 5) and is async-only: it is honored by router.acompletion() and other async entrypoints, while the sync router.completion() path falls through to regular fallbacks. Cooldowns still apply independently. [source]
If there is insufficient available memory to load a new model request while one or more models are already loaded, all new requests will be queued until the new model can be loaded. As prior models become idle, one or more will be unloaded to make room for the new model. Queued requests will be processed in order. When using GPU inference new models must be able to completely fit in VRAM to allow concurrent model loads. [source]
In this case, it's more convenient to treat each DP rank like a separate vLLM deployment, with its own endpoint, and have an external router balance HTTP requests between them, making use of appropriate real-time telemetry from each server for routing decisions. [source]
By default, when a deployment in a model group fails, the router moves on to the next entry in fallbacks, which is a different model group. With enable_weighted_failover, the router first retries inside the same model group by re-picking a different deployment using the existing weights, and only escalates to cross-group fallbacks once every deployment in the group has been tried. This is useful for multiple regional copies of the same model, where a failed region should fail over to a healthy peer with the same model_name. [source]
When deploying large DP sizes using this method, the API server process can become a bottleneck. In this case, the orthogonal --api-server-count command line option can be used to scale this out (for example --api-server-count=4). This is transparent to users - a single HTTP endpoint / port is still exposed. Note that this API server scale-out is "internal" and still confined to the "head" node. [source]
When a single DP group requires multiple nodes, e.g. in case a single model replica needs to run on at least two nodes, make sure to set VLLM_RAY_DP_PACK_STRATEGY="span" in which case --data-parallel-size-local is ignored and will be automatically determined [source]
Remote DP ranks will be allocated based on node resources of the Ray cluster [source]
Ollama supports two levels of concurrent processing. If your system has sufficient available memory (system memory when using CPU inference, or VRAM for GPU inference) then multiple models can be loaded at the same time. For a given model, if there is sufficient available memory when the model is loaded, it is configured to allow parallel request processing. [source]
Parallel request processing for a given model results in increasing the context size by the number of parallel requests. For example, a 2K context with 4 parallel requests will result in an 8K context and additional memory allocation. [source]

Parameters and configuration

var ( // NumParallel sets the number of parallel model requests. NumParallel can be configured via the OLLAMA_NUM_PARALLEL environment variable. NumParallel = Uint("OLLAMA_NUM_PARALLEL", 1) // MaxRunners sets the maximum number of loaded models. MaxRunners can be configured via the OLLAMA_MAX_LOADED_MODELS environment variable. MaxRunners = Uint("OLLAMA_MAX_LOADED_MODELS", 0) // MaxQueue sets the maximum number of queued requests. MaxQueue can be configured via the OLLAMA_MAX_QUEUE environment variable. MaxQueue = Uint("OLLAMA_MAX_QUEUE", 512) // MaxTransferStreams caps the number of simultane [source]
* `OLLAMA_MAX_LOADED_MODELS` - The maximum number of models that can be loaded concurrently provided they fit in available memory. The default is 3 \* the number of GPUs or 3 for CPU inference. * `OLLAMA_NUM_PARALLEL` - The maximum number of parallel requests each model will process at the same time, default 1. Required RAM will scale by `OLLAMA_NUM_PARALLEL` \* `OLLAMA_CONTEXT_LENGTH`. * `OLLAMA_MAX_QUEUE` - The maximum number of requests Ollama will queue when busy before rejecting additional requests. The default is 512 [source]
router_settings routing_strategy accepts simple-shuffle, least-busy, usage-based-routing or latency-based-routing, and defaults to simple-shuffle. router_settings also takes num_retries (for example 2) and timeout (for example 30 seconds). [source]

How-to and procedures

For larger scale deployments especially, it can make sense to handle the orchestration and load balancing of data parallel ranks externally. [source]
Set redis_host, redis_port and redis_password in router_settings when you run multiple LiteLLM proxy deployments; load balancing state is then stored in Redis. [source]
Set allowed_fails and cooldown_time on the Router. A deployment that fails more than allowed_fails calls in a minute is cooled down for cooldown_time seconds (for example allowed_fails=1 and cooldown_time=100 cool a deployment for 100 seconds). [source]
Set routing_strategy to least-busy to route each call to the deployment with the fewest active requests. [source]

Examples and snippets

It can be configured by simply including e.g. --data-parallel-size=4 in the vllm serve command line arguments. This will require 4 GPUs. It can be combined with tensor parallel, for example --data-parallel-size=4 --tensor-parallel-size=2, which would require 8 GPUs. When sizing DP deployments, remember that --max-num-seqs applies per DP rank. [source]
This will run DP=4 with DP ranks 0 and 1 on the head node and ranks 2 and 3 on the second node: [source]

Measurements and reference values

Results: vLLM performed up to 3.23x faster than Ollama with 128 concurrent requests (vLLM reached 70.69 requests/s at 128 concurrent; average latency 1.754 s, p95 2.482 s). Ollama ramped up to about 22 requests per second with 32 concurrent requests (matching its OLLAMA_NUM_PARALLEL=32) and stayed flat at that level across the rest of the tests; adding more concurrent requests beyond 32 only increased latency. [source]
Hardware: 2x AMD EPYC 7343, 256 GB DDR5, 2x NVIDIA A6000 48 GB with NVLink (96 GB VRAM). Models: vLLM Qwen/Qwen3-14B (vLLM API server 0.8.5.post1, launched with --tensor-parallel-size 2); Ollama qwen3:14b-fp16 (version not stated). Ollama configuration: OLLAMA_SCHED_SPREAD=1 CUDA_VISIBLE_DEVICES=0,1 OLLAMA_NUM_PARALLEL=32 ollama serve. [source]
Abstract: Serving LLMs to many concurrent users gives rise to significant challenges for latency, throughput, and scalability. This paper presents a systematic and reproducible benchmark of two widely used LLM serving frameworks, Ollama and vLLM, under concurrent workloads. We conducted a controlled comparison by running both frameworks on a single NVIDIA H100 80 GB GPU with the same model weights (Qwen3-4B) and inference configurations. We then evaluated them using five open benchmark datasets across four scenarios consisting of baseline question answering, complex reasoning, streaming intera [source]
Baseline scenario results table: Concurrency 10: Ollama avg latency 20.92 s, 0.45 req/s, 0% errors; vLLM 3.71 s, 2.61 req/s, 0% errors. Concurrency 50: Ollama 89.72 s avg, 0.46 req/s, 0% errors; vLLM 4.02 s, 11.93 req/s. Concurrency 100: Ollama 100.48 s avg, P95 126.09 s, 0.45 req/s, 39.29% errors; vLLM 6.37 s avg, P95 7.94 s, 15.20 req/s, 0% errors. Ollama throughput plateaus at 0.45-0.51 req/s in every scenario; vLLM holds 9.91-13.33 req/s depending on the workload. Ollama error rate ranges from 13.1% (baseline) to 30.06% (streaming). [source]
Stress test: Ollama breaking point at 10 users; degradation zone 10-25 users; dominant failure: throughput stagnation, queue saturation, extended recovery after spikes. vLLM stable to 100 users, degradation 100-200, breaking point >= 200 users (15.46 req/s at 200, avg 12.23 s). Endurance: vLLM completed 56,345 requests at 15.65 req/s; Ollama 1,722 requests at 0.48 req/s. [source]

Problems, failure modes and limitations

If too many requests are sent to the server, it will respond with a 503 error indicating the server is overloaded. You can adjust how many requests may be queued by setting `OLLAMA_MAX_QUEUE`. [source]
Note: Windows with Radeon GPUs currently default to 1 model maximum due to limitations in ROCm v5.7 for available VRAM reporting. Once ROCm v6.2 is available, Windows Radeon will follow the defaults above. You may enable concurrent model loads on Radeon on Windows, but ensure you don't load more models than will fit into your GPU's VRAM. [source]

Comparisons and alternatives

A single launch command (on any node) is needed to start all local and remote DP ranks, therefore it is more convenient compared to launching on each node [source]

Facts and statements

LiteLLM automatically distributes requests across multiple deployments of the same model using its built-in router. The proxy routes traffic to optimize performance and reliability. The "simple-shuffle" routing strategy is used by default. [source]
Running a single data parallel deployment across multiple nodes requires a different vllm serve to be run on each node, specifying which DP ranks should run on that node. In this case, there will still be a single HTTP entrypoint - the API server(s) will run only on one node, but it doesn't necessarily need to be co-located with the DP ranks. [source]
Load balance multiple instances of the same model. The LiteLLM proxy handles routing requests using LiteLLM's Router. Set rpm in the config if you want to maximize throughput. [source]
Method: same hardware, same model, concurrency steps 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 768, 1000; throughput in requests per second and response latency. [source]
least-busy routes to the deployment with the fewest active requests. Use it for high concurrency scenarios. [source]
The LiteLLM Router provides multiple strategies for routing calls across multiple deployments: Weighted Pick, Rate-Limit Aware v2, Latency-Based, Rate-Limit Aware, Least-Busy, Custom Routing Strategy and Lowest Cost Routing. We recommend using simple-shuffle (the default) for best performance in production. [source]
Setup: Ollama 0.24.0 serving qwen3:4b-q8_0 (GGUF, Q8_0) with OLLAMA_NUM_PARALLEL=4; vLLM 0.17.0 serving Qwen/Qwen3-4B with native FP8. Single node, single GPU. [source]
In the above diagram, each of the dotted boxes corresponds to a separate launch of vllm serve - these could be separate Kubernetes pods, for example. [source]
Give a deployment a weight to pick it more often than others that share the same model_name. For example weight 2 picks that deployment twice as often as a deployment with weight 1. [source]
simple-shuffle (recommended) randomly distributes requests. Use it for general purpose work and even load distribution. [source]
usage-based-routing (documented as bad for perf) routes to the deployment with the lowest current usage (RPM/TPM). Use it when you want to respect rate limits evenly. [source]
latency-based-routing routes to the fastest responding deployment. Use it for latency-critical applications. [source]
cost-based-routing routes to the deployment with the lowest cost. Use it for cost-sensitive applications. [source]
LiteLLM supports all models from Ollama. We recommend using ollama_chat for better responses. [source]
Usage-based routing adds significant latency due to Redis operations for tracking usage across deployments. The async implementation filters out a deployment if its tpm/rpm limit is exceeded, and routes to the deployment with the lowest TPM usage for that minute. In production, Redis tracks TPM/RPM across multiple deployments. [source]

Related concepts

vLLM — is a contrast of Ollama scale-out and vLLM
DP rank — is a part of Ollama scale-out and vLLM
parallel requests — is a part of Ollama scale-out and vLLM
simple-shuffle — is a instance of Ollama scale-out and vLLM
load balancer — is a part of Ollama scale-out and vLLM
data parallel — is a part of Ollama scale-out and vLLM
queued — is a problem of Ollama scale-out and vLLM
weighted failover — is a part of Ollama scale-out and vLLM
OLLAMA_NUM_PARALLEL — is a part of Ollama scale-out and vLLM
LiteLLM — is a instance of Ollama scale-out and vLLM
throughput — is a measure of Ollama scale-out and vLLM
benchmark — is a measure of Ollama scale-out and vLLM
OLLAMA_MAX_QUEUE — is a part of Ollama scale-out and vLLM
tensor parallel — is a contrast of Ollama scale-out and vLLM
OLLAMA_MAX_LOADED_MODELS — is a part of Ollama scale-out and vLLM
external load balancer — is a part of Ollama scale-out and vLLM
KV cache — is a related of Ollama scale-out and vLLM
OLLAMA_SCHED_SPREAD — is a part of Ollama scale-out and vLLM
replica — is a part of Ollama scale-out and vLLM
prefix-aware routing — is a part of Ollama scale-out and vLLM

Children

No children recorded.

← the whole tree · 3D view· how to read this page