Long-running secondary reads in sharded clusters

Published reference · snapshot 2026-10-20 · skill mongodb-expert/references/long-running-secondary-reads-in-sharded-clusters.md

↓ Facts as markdown all context files

Source-anchored research for Long-running secondary reads in sharded clusters. Run status: COMPLETED.

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Structure and components

Monitor termination with db.serverStatus().metrics.operation.killedDueToRangeDeletion on shard secondaries. The counter was added in 8.2 and counts operations terminated because of orphan range cleanup. Each termination also writes a mongod log entry with id 10016300, component SHARDING, and message 'Read has been terminated due to orphan range cleanup'. The entry's attr includes orphanCleanupDelaySecs. (high confidence) [source]

How it works

Since MongoDB 8.2, the mongod parameter terminateSecondaryReadsOnOrphanCleanup defaults to true. With it on, a secondary terminates reads that began before the chunk migration committed and are still running when the secondary is about to apply the donor's orphan deletion. The rule covers queries, aggregations and operations that span several namespaces. (high confidence) (verified as of 2026-10-20) [source]
terminateSecondaryReadsOnOrphanCleanup is a boolean per mongod. It can be set at startup or at runtime with setParameter. Because it is per mongod, different members can hold different values. (medium confidence) [source]
SERVER-100158 defines the kill rule. The server kills a query with QueryPlanKilled when three conditions hold: the Range Preserver tied to the query has been invalidated, the read concern is neither 'snapshot' nor 'available', and the feature flag killReadsDueToOrphanCleanup is on. The manual instead says the behavior applies to all read operations on secondaries (see disagreements). (low confidence) (verified as of 2026-10-20) [source]
SERVER-132078, fixed in 9.0.4 and 9.1.0-rc0, says a chunk moved away from a shard may not be movable back until minSnapshotHistoryWindowInSeconds (default 5 minutes) has passed, because the migration must keep the recipient's point-in-time snapshot history. The ticket says production rarely sees this because moving a chunk back is already delayed by orphanCleanupDelaySecs (1 hour). Inference, not stated in the ticket: a snapshot window set above orphanCleanupDelaySecs would become the binding delay. (low confidence) (verified as of 2026-10-20) [source]
MongoDB lists the resume mechanism first and offers three alternatives: raise orphanCleanupDelaySecs, disable secondary read termination, or disable the balancer. It advises testing the resume mechanism in a test environment and monitoring production for how often secondary reads terminate. If terminations are frequent, adjust query patterns or consider other data access approaches. (low confidence) [source]
Do not run migrations with _waitForDelete set to true on clusters that serve long secondary reads. The option removes the orphanCleanupDelaySecs wait before range deletion. The 8.2 manual warns that secondary reads then might terminate because of the delete phase. SERVER-87673 says that before the 8.2 fix any secondary read could miss documents after such a migration, and it describes the option as meant for internal testing. (medium confidence) [source]
Setting terminateSecondaryReadsOnOrphanCleanup to false restores the pre-8.2 behavior. Reads run to completion but can silently miss documents when chunk migrations touch the collection. The manual presents the setting only as a way to match MongoDB 8.1 and earlier, and it warns that secondary reads might not return all documents. (high confidence) [source]
Severalnines (published 2016, updated 2022) says reads from secondaries on a sharded cluster can return incomplete data, because chunks may be in transit during a migration, and says never to bypass the shard router (mongos). (low confidence) [source]
Raising minSnapshotHistoryWindowInSeconds (for example to 600 with setParameter) increases disk usage because the server keeps more history. On Atlas, only Atlas Support can change it. In sharded clusters, changing it on config server nodes may break internal operations, and setting it to 0 there breaks internal snapshot reads against the config server. (high confidence) (verified as of 2026-10-14) [source]
When a secondary's WiredTiger cache is under pressure, application threads evict pages, fewer threads remain for replication, and incoming queries must evict cache before they run, which can raise replication lag. MongoDB's guide says to check whether db.serverStatus().wiredTiger.cache['pages evicted by application threads'] is above 0 and rising. Netdata's guide independently lists long-running aggregations and operations in db.currentOp() that hold tickets or locks as a cause of secondary lag. (medium confidence) [source]
SERVER-34192 'Secondary reads during batch applications' (a New Feature fixed in 3.7.4, the development series before MongoDB 4.0) is the engineering change that lets a secondary serve reads while it applies replication batches. SERVER-20328 'Allow secondary reads while applying oplog entries' closed as its duplicate. The ticket page did not show a description, so the exact mechanism is unverified here. (low confidence) [source]
MongoDB lists three alternatives to a resume mechanism. First, raise orphanCleanupDelaySecs at runtime, for example setParameter orphanCleanupDelaySecs: 7200. Second, set terminateSecondaryReadsOnOrphanCleanup to false and accept that reads might silently miss documents. Third, avoid migrations entirely: disable balancing for the collection (configureCollectionBalancing with enableBalancing) or restrict the balancing window, and run no manual migrations. The docs warn that long balancer pauses can leave shards unbalanced. (high confidence) (verified as of 2026-10-14) [source]
A new orphanCleanupDelaySecs value applies only to range deletions created after the change. To apply it to range deletions already queued, force a step down. SERVER-96417 (Backlog and unresolved when fetched on 2026-10-20) reports that the value is not persisted across step downs: after a step-down or step-up the parameter returns to its original value, and queued deletions wait the full delay again. (medium confidence) (verified as of 2026-10-20) [source]
A snapshot read with an atClusterTime older than the oldest retained snapshot fails (SnapshotTooOld). A snapshot read with an atClusterTime the member has not reached yet waits, so bound the wait with maxTimeMS when reads target lagging members. (medium confidence) [source]
maxStalenessSeconds lets a client skip secondaries whose estimated lag exceeds the value. The minimum is 90 seconds. It applies only when a secondary is selected for a read, and mongos re-evaluates read preference for every operation. MongoDB lists long-running operations among the causes of secondary lag. (medium confidence) [source]
MongoDB 8.2.2 (2025-11-19) fixed SERVER-105047: queries using $or and $ne on sharded collections might include orphan documents. This scopes the assumption that read concern 'local' always filters orphans. Check that your 8.2 build includes the fix when exact results matter. (low confidence) [source]

How-to and procedures

To run related reads against one consistent point in time on secondaries, start a session with snapshot enabled (for example start_session(snapshot=True)) and run every query in that session. Transactions that contain reads must use read preference primary, so snapshot reads outside transactions are the way to get snapshot isolation on secondaries. (medium confidence) [source]

Measurements and reference values

mongos applies the read preference to each replica-set shard and re-evaluates it for every operation. For secondary reads it picks randomly among eligible members within the latency window of the closest member (15 ms by default). Different operations in one job can therefore land on different secondaries. MongoDB suggests causally consistent sessions to get monotonic reads. (medium confidence) [source]

Problems, failure modes and limitations

A sharded query follows the shard-version protocol: it reads each document from the shard that owned the document when the query began. The donor shard deletes migrated documents only after its own primary has no query left on the old placement and orphanCleanupDelaySecs has passed. The donor primary cannot see queries that run on its secondaries. A secondary read that outlives that delay can therefore miss documents it still needs. (high confidence) [source]
Only secondary reads that start before the chunk migration commits are exposed. A read that starts after the commit does not use the migrating chunk. Before MongoDB 8.2, an exposed read that kept running after the donor deleted the orphans returned incomplete results and no error. (high confidence) [source]
MongoDB's recommended fix for secondary reads that run longer than one hour on clusters that migrate chunks is an application-side resume mechanism. The application sorts on an indexed field with unique values. It catches error 175 whose errmsg is the orphan-range-cleanup message. It then re-runs the query with {sortField: {$gt: lastProcessedValue}}. If the sort field is not unique, the application must handle ties. A compound index on the filter field plus the sort field avoids collection scans and in-memory sorts. The Alibaba MongoShake full-sync reader uses the same pattern: it rebuilds a killed cursor with {_id: {$gt: lastReadID}} plus the original range. (medium confidence) [source]
Raising orphanCleanupDelaySecs lets secondary reads run longer before the donor deletes orphans. The parameters page says to consider raising it when queries on shard secondaries exceed 60 minutes. The cost is that orphans stay on the donor longer. A query that uses an index but omits the shard key then filters more orphans and can slow down. A shard with storage limits can lower the value temporarily. (high confidence) [source]
Stopping chunk migrations removes the trigger. The manual lists three ways: disable the balancer and run no manual migrations, disable balancing per collection with configureCollectionBalancing enableBalancing, or limit the balancer to a balancing window. It warns that long periods without balancing leave shards unbalanced. Percona Backup for MongoDB applies the same idea to sharded backups: it stops the balancer before a backup starts and fails the backup if the balancer does not stop within backup.timeouts.balancerStop (default 0, no limit). (medium confidence) [source]
On MongoDB Atlas, customers cannot change minSnapshotHistoryWindowInSeconds themselves. They must contact Atlas Support. (medium confidence) (verified as of 2026-10-20) [source]
On Atlas, send long-running or ETL reads to analytics nodes with readPreference=secondary and readPreferenceTags=nodeType:ANALYTICS. Add a final empty tag set (readPreferenceTags=) as a fallback. Without a fallback, analytics reads can fail during initial sync after data compaction, during disk scale-down and during NVMe cluster tier changes. The read preference manual explains why: the client tries tag sets in order and the read errors when no secondary matches. (medium confidence) (verified as of 2026-10-18) [source]
In sharded clusters, changing minSnapshotHistoryWindowInSeconds on config server nodes may make internal operations fail. Do not set it to 0 on config server nodes, because internal snapshot reads that target the config server with an atClusterTime then fail. (low confidence) [source]
A read that an 8.2 secondary terminates fails with QueryPlanKilled, code 175, category CursorInvalidatedError, errmsg 'Read has been terminated due to orphan range cleanup'. The server also logs that message in component SHARDING (log id 10016300) with the orphanCleanupDelaySecs value. MongoDB calls the error not retryable by design. (high confidence) [source]
QueryPlanKilled (175) also fires for other causes, so match the errmsg and not only the code. The manual says so. In the MongoShake review, dropping a hinted index under an open cursor returned code 175 with the message '(QueryPlanKilled) query plan killed :: index sk_1 dropped'. Code 237 is a different error, CursorKilled. (medium confidence) [source]
MongoDB removes an idle cursor after cursorTimeoutMillis, default 600000 ms (10 minutes). A client that pauses between getMore calls for longer than that loses the cursor. This failure is separate from orphan-cleanup termination. (low confidence) [source]
Through MongoDB 8.1, the server does not terminate long-running secondary reads. A read that began on a secondary before a chunk migration keeps running after the orphan deletions replicate, and it can silently miss documents without returning an error. In 8.2 and later, setting terminateSecondaryReadsOnOrphanCleanup to false restores this behavior. (high confidence) (verified as of 2026-10-14) [source]
A secondary cannot defer a delete that arrives in the oplog, so the primary-side delay is the only grace period a long-running secondary read gets. SERVER-29405 (fix version 3.5.10, in the development series that became 3.6) added that delay with a 15-minute default. SERVER-87673 records that queries on secondaries running longer than the delay could miss documents in versions 3.6 through 8.0. It closed as a duplicate of SERVER-100158, the 8.2 fix. (high confidence) [source]
Outside multi-document transactions, read concern 'snapshot' works on primaries and secondaries from MongoDB 5.0 for find, aggregate, and distinct (distinct on unsharded collections only). A snapshot read must finish inside the storage engine's history window, minSnapshotHistoryWindowInSeconds, which defaults to 300 seconds. A session or query that outlasts the window fails with a SnapshotTooOld error. (high confidence) (verified as of 2026-10-14) [source]
MongoDB's recommended handling is an application-level resume mechanism. Sort the query on a unique, indexed field and keep the last processed value. When the read fails with code 175 and errmsg 'Read has been terminated due to orphan range cleanup', re-issue the query with a $gt filter on the last value. Match on errmsg, because code 175 (QueryPlanKilled) also occurs for other reasons. The parameters page states that this error is not retryable by design. (high confidence) [source]
A longer orphanCleanupDelaySecs keeps orphaned documents on the donor shard for longer. Index queries that omit the shard key must then filter more orphans and can slow down. The parameters page also advises lowering the value temporarily when a shard has storage constraints, and raising it when queries on shard secondaries run longer than 60 minutes. (high confidence) [source]
On Atlas, analytics nodes carry the tag nodeType:ANALYTICS. The documented connection options for long ETL or reporting reads are readPreference=secondary, readPreferenceTags=nodeType:ANALYTICS, and readConcernLevel=local. Atlas recommends adding a final empty tag set (readPreferenceTags=) as a fallback, because analytics-node queries can fail during initial sync after compaction, disk scale-down, or NVMe tier changes. The Atlas examples use readConcernLevel=local so secondary reads on sharded clusters do not return orphaned documents. (medium confidence) (verified as of 2026-10-18) [source]
MongoDB says secondary reads that exceed 1 hour on sharded clusters that perform chunk migrations might fail with QueryPlanKilled (175). The 1-hour figure is the 8.2 default for orphanCleanupDelaySecs, and the primary-side wait for in-progress reads comes before it. If a long job fails with the orphan-cleanup errmsg, compare its runtime with that window and check killedDueToRangeDeletion. (medium confidence) (verified as of 2026-10-14) [source]

Comparisons and alternatives

In a sharded cluster, mongos cannot reach hidden members, and MongoDB says reading from them by direct connection can cause data inconsistency or loss. MongoDB's instruction for workload isolation is tag-based read preferences. Tag the dedicated secondaries on each shard and send long reads with readPreference=secondary plus readPreferenceTags, rather than secondaryPreferred. (high confidence) [source]
Run $currentOp on mongos with targetAllNodes: true (new in 7.1) to get one document per data-bearing node instead of one per shard. For three shards of three nodes each, false returns three documents and true returns nine. This exposes long-running reads on secondaries without connecting to each node. (medium confidence) [source]

Facts and statements

MongoDB tracked this silent-miss defect as SERVER-87673. The ticket lists affected versions 3.6.0, 4.0.0, 4.2.0, 4.4.0, 5.0.0, 6.0.0, 7.0.0 and 8.0.0-rc0. Jira links SERVER-87673 as fixed by SERVER-100158, whose fix version is 8.2.0-rc0. (medium confidence) [source]
SERVER-87673 names a second exposure path. A query that starts on a primary and keeps running after that member steps down becomes a secondary read. If it then runs longer than orphanCleanupDelaySecs after a migration, it can miss documents the same way. (low confidence) [source]
orphanCleanupDelaySecs defaults to 900 seconds (15 minutes) in the MongoDB 8.0 manual and to 3600 seconds (1 hour) from MongoDB 8.2. A new value applies only to range deletions created after the change. To apply it to existing range deletions, force a step down. (high confidence) (verified as of 2026-10-20) [source]
The default read concern for reads against secondaries is 'local'. Secondaries filter orphaned documents for 'local' reads. They do not filter them for 'available' reads. (high confidence) [source]
Every member of a shard replica set keeps chunk metadata, so secondary reads do not return orphaned documents unless they use read concern 'available'. Earlier versions returned orphans from secondaries whatever the read concern. Read concern 'available' skips the ownership check and can return orphaned documents while a chunk migrates. Atlas tells users to add readConcernLevel=local to connection strings for secondary reads on sharded clusters. (high confidence) [source]
MongoDB tracked the older problem as SERVER-5931, 'Secondary reads in sharded clusters need stronger consistency'. The ticket says duplicate, stale or missing documents can appear on secondary reads while the balancer runs. Jira marks it Done with fix version 3.5.11, resolved 2017-07-31. The current manual says earlier versions could return orphans from secondaries regardless of read concern. (medium confidence) [source]
Read concern 'snapshot' works on secondaries since MongoDB 5.0 for find and aggregate, and for distinct on unsharded collections only. A session started with snapshot=true gives every query in it the same point in time. The default history window is 300 seconds. A snapshot query or session that lasts longer fails with SnapshotTooOld. Raising minSnapshotHistoryWindowInSeconds lengthens the window (the manual's example sets 600) and costs extra disk space for history. (high confidence) (verified as of 2026-10-14) [source]
A resume bound must follow the cursor's real sort key. The first MongoShake fix wrote an _id resume point into a shard-key range while the cursor sorted by _id. BSON type bracketing made the rebuilt query return zero documents. A reviewer's live killCursors test on MongoDB 7.0.37 showed that version resumed 0 documents and silently lost 899. The merged version keeps the original range and adds {_id: {$gt: lastReadID}}. (low confidence) [source]
Count terminations with db.serverStatus().metrics.operation.killedDueToRangeDeletion on each secondary. MongoDB added the counter in 8.2. Each termination also writes a mongod log line with the message 'Read has been terminated due to orphan range cleanup'. (high confidence) [source]
From MongoDB 8.2, the server terminates a secondary read in a sharded cluster if the read began before a chunk migration committed and the donor shard is about to delete the migrated documents. The client receives QueryPlanKilled (code 175, category CursorInvalidatedError) with errmsg 'Read has been terminated due to orphan range cleanup'. The mongod parameter terminateSecondaryReadsOnOrphanCleanup controls this behavior and defaults to true. (high confidence) (verified as of 2026-10-14) [source]
After a chunk migration commits, the donor shard's primary waits for in-progress reads on the migrated namespace to finish on that primary. It then waits a further orphanCleanupDelaySecs and deletes the orphaned documents. Secondaries replay those deletes from the oplog. MongoDB 8.2 raised the orphanCleanupDelaySecs default from 900 seconds (15 minutes) to 3600 seconds (1 hour). (high confidence) (verified as of 2026-10-14) [source]
The 8.2 termination applies only to reads that began before the migration commit. It covers queries, aggregations, and operations that span multiple namespaces. The server terminates the read before the secondary removes the orphaned documents. (medium confidence) (verified as of 2026-10-14) [source]
From the 3.6 development series (SERVER-5931, fix version 3.5.11), every member of a shard's replica set holds chunk metadata, so a secondary read with read concern 'local' (the default) filters out orphaned documents. Read concern 'available' skips that filtering and can return orphans. Before chunk-aware secondaries, secondary reads could return orphans under any read concern. (high confidence) [source]
SERVER-100158 (fix version 8.2.0-rc0) states the kill rule as three conditions: the query's range preserver was invalidated, the operation's read concern is other than 'snapshot' or 'available', and the feature flag killReadsDueToOrphanCleanup is enabled. If the shipped behavior matches this rule, reads with read concern 'snapshot' or 'available' are not terminated by orphan cleanup. (low confidence) [source]
SERVER-31837 describes orphanCleanupDelaySecs as having the same effect as a cursor held open for 15 minutes on the donor shard's primary, and says a recipient shard that receives a chunk range back could wait up to that long. The ticket closed as 'Gone away' on 2025-04-09 with no fix version, so whether the recipient wait still exists in 8.x is unverified. (low confidence) (verified as of 2026-10-20) [source]
A long read needs a cursor and session that outlive the batch-processing time. In a sharded cluster, maxTimeMS counts mongos-to-mongod latency but not client-to-server latency. A session idle for more than 30 minutes expires, and the server then kills its cursors even if they use noCursorTimeout() or a maxTimeMS above 30 minutes. Use an explicit session and call refreshSessions periodically. Cursors tied to a session follow localLogicalSessionTimeoutMinutes, not cursorTimeoutMillis. (medium confidence) [source]
If a snapshot read does not supply atClusterTime, the mongos (or the mongod in a single-member replica set) picks the latest majority-committed snapshot timestamp and returns it to the client. (low confidence) [source]
Alibaba Cloud ApsaraDB for MongoDB offers read-only nodes for replica set and sharded cluster instances, up to five per shard, each with its own connection string URI. They replicate asynchronously, with lag usually in milliseconds and up to seconds under heavy primary writes. Alibaba positions them for analytics and BI reads. (low confidence) (verified as of 2026-10-20) [source]
Since MongoDB 8.0, you can run only a listed set of commands directly on a node in a sharded cluster. The list includes currentOp and killOp but no find or aggregate on user collections, and the page says it might not be exhaustive. Running a long read straight on a shard secondary is therefore not a supported workaround; route the read through mongos with tags. (medium confidence) (verified as of 2026-10-14) [source]
For dedicated ETL or reporting reads, MongoDB prefers read preference secondary over secondaryPreferred. If every secondary is unavailable, secondaryPreferred sends all traffic to the primary, where the queries compete with writes. (low confidence) [source]
A forum user (2025-06-12) reports three limits of $currentOp targetAllNodes. It fails on a plain replica set with 'supports targetAllNodes parameter only for sharded clusters'. On a sharded cluster it fails with HostUnreachable 'node is not in primary or recovering state' if any secondary is in RECOVERING. Without the option, mongos returns only each shard's primary. The user fell back to running $currentOp on each mongod. (low confidence) (verified as of 2025-06-12) [source]
The serverStatus counter rangeDeleterTasks (shard members only) counts queued range deletion tasks that are ready to run or running. The config.rangeDeletions collection lists the chunk ranges pending deletion from a shard after a migration. (low confidence) [source]
Count orphans with $shardedDataDistribution. It runs on mongos only, on the admin database, and needs the shardedDataDistribution privilege action. It returns shards.numOrphanedDocs and shards.orphanedSizeBytes per shard, and MongoDB documents it for confirming that no orphans remain from 6.0.3. Huawei Cloud's FAQ describes an older check: run find().readPref('secondary').readConcern('local').explain('executionStats') through mongos and read chunkSkips in the SHARDING_FILTER stage. Huawei warns that this scans every document on every shard node and can load the instance. (medium confidence) [source]
Do not use secondary or secondaryPreferred to add read capacity. All replica set members take roughly the same write traffic, replication is asynchronous so secondary reads can be stale and non-monotonic, and losing a member can overload the rest. MongoDB says sharding is often the better strategy for capacity. (high confidence) [source]
Do not run migrations with _waitForDelete: true while long secondary reads are in flight. It skips the orphanCleanupDelaySecs wait, so range deletion starts right after the migration and running secondary reads can be terminated (8.2 and later) or miss documents (8.1 and earlier). MongoDB documents _waitForDelete as meant for internal testing. (high confidence) [source]
Separate mongos pools per application isolate only the routers. Alibaba Cloud's sharded-cluster guide says the backend shards remain shared, so a long analytics job routed through a dedicated mongos pool still reads from the same shard members as other traffic. (low confidence) [source]
Long reads on the donor shard's primary delay orphan cleanup. MongoDB waits for in-progress queries involving the chunk to finish on the primary before it starts the orphanCleanupDelaySecs wait, and range deletion may not proceed while there are ongoing reads or open cursors on the collection. (medium confidence) [source]

Children

Atlas analytics nodes and read preference tags (frontier)
Balancer control for long reads (balancing window, configureCollectionBalancing, _waitForDelete) (frontier)
Balancer windows and per-collection balancing control (frontier)
Chunk migration commit, range deleter and orphan cleanup internals (frontier)
Keyset (resume) pagination for restartable cursors (frontier)
Orphaned documents and read concern available versus local (frontier)
Resume-from-last-key pattern for killed cursors (frontier)
Snapshot reads (atClusterTime) and history window sizing on secondaries (frontier)
Snapshot reads on secondaries and minSnapshotHistoryWindowInSeconds (frontier)
Workload isolation with tagged secondaries and Atlas analytics nodes (frontier)
orphanCleanupDelaySecs and range deletion timing (frontier)
terminateSecondaryReadsOnOrphanCleanup and QueryPlanKilled 175 (frontier)

Frontier under this node: Atlas analytics nodes and read preference tags, Balancer control for long reads (balancing window, configureCollectionBalancing, _waitForDelete), Balancer windows and per-collection balancing control, Chunk migration commit, range deleter and orphan cleanup internals, Keyset (resume) pagination for restartable cursors, Orphaned documents and read concern available versus local, Resume-from-last-key pattern for killed cursors, Snapshot reads (atClusterTime) and history window sizing on secondaries, Snapshot reads on secondaries and minSnapshotHistoryWindowInSeconds, Workload isolation with tagged secondaries and Atlas analytics nodes, orphanCleanupDelaySecs and range deletion timing, terminateSecondaryReadsOnOrphanCleanup and QueryPlanKilled 175

← the whole tree · 3D view· how to read this page