Shard draining cancellation, status and commit protocol
Parent: Sharded Cluster Cost Economics · Published reference · snapshot 2026-10-29
↓ Facts as markdownall context files
Source-anchored research for Shard draining cancellation, status and commit protocol. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Definitions
- checkMetadataConsistency is a separate sharding-catalog diagnostic, available only through mongos. It supports cluster, database and collection scope, returns a cursor of inconsistencies, and excludes index consistency checks unless checkIndexes is enabled. These checks do not by themselves certify user-document integrity. (low confidence) (verified as of 2026-10-14) [source]
Structure and components
- The removal tutorial says an in-progress drain survives cluster restarts because MongoDB records draining state in config.shards and automatically continues draining after components restart. Restarting the cluster is therefore not a documented cancellation mechanism. (low confidence) (verified as of 2026-10-14) [source]
How it works
- MongoDB documents majority write concern for startShardDraining and says it waits for concurrent DDL to finish. The removal tutorial also warns that creating, sharding or moving collections during the procedure can cause interruptions and unexpected results. (medium confidence) [source]
- checkMetadataConsistency is a separate mongos diagnostic that returns a cursor of catalog inconsistencies. It does not check indexes by default; use checkIndexes: true when index consistency is required. The documented shard-removal procedure does not make this diagnostic a mandatory commit step. (medium confidence) [source]
- MongoDB documents that startShardDraining waits for concurrent DDL operations to finish. The removal tutorial separately warns that creating, sharding or moving collections during the procedure may cause interruptions and unexpected results; the staged API should not be treated as a general concurrency-safety guarantee. (medium confidence) (verified as of 2026-10-14) [source]
How-to and procedures
- Run db.adminCommand({startShardDraining: "shard04"}) through mongos to request asynchronous chunk evacuation. An ok response acknowledges the request and does not establish that evacuation or removal has completed. (medium confidence) [source]
- Use db.adminCommand({shardDrainingStatus: "shard04"}) to inspect state, remaining chunks, database count, jumbo chunks, unsharded collections and estimated remaining bytes, together with dbsToMove and collectionsToMove. Polling status is distinct from committing removal. (medium confidence) [source]
Problems, failure modes and limitations
- stopShardDraining returns an error for a nonexistent shard and otherwise returns ok. (low confidence) [source]
- Treat movePrimary as a separate maintenance boundary. MongoDB warns against reads or writes to affected unsharded collections during that operation and describes migration failure or data-loss risk; move unsharded collections first. (low confidence) [source]
Comparisons and alternatives
- MongoDB documents startShardDraining, stopShardDraining, shardDrainingStatus and commitShardRemoval as new in 8.3; removeShard is deprecated from 8.3 rather than already removed. The staged interface separates starting, cancelling, observing and committing removal. (medium confidence) (verified as of 2026-10-14) [source]
Facts and statements
- The four staged commands support self-managed MongoDB Community and Enterprise deployments; the command references explicitly exclude MongoDB Atlas and direct Atlas users to its managed shard-scaling workflow. (medium confidence) (verified as of 2026-10-14) [source]
- The balancer must be enabled to migrate chunks off a draining shard. Starting a drain while it is disabled leaves remaining work visible through shardDrainingStatus. (medium confidence) [source]
- db.adminCommand({stopShardDraining: "shard04"}) cancels the draining intent while retaining the shard. MongoDB then includes the shard in normal rebalancing and can move chunks back onto it. Cancellation therefore does not promise to preserve the data placement reached before the stop. (medium confidence) (verified as of 2026-10-14) [source]
- MongoDB 8.3 adds listShards filter.draining: true selects draining shards, false selects non-draining shards, and an omitted filter returns all shards. listShards runs on mongos against admin. (medium confidence) (verified as of 2026-10-14) [source]
- Move each unsharded collection in collectionsToMove with moveCollection before moving the database primary with movePrimary for entries in dbsToMove. Zero remaining sharded chunks alone does not satisfy the removal prerequisites. (medium confidence) (verified as of 2026-10-14) [source]
- After shardDrainingStatus reports state: "drainingComplete", run db.adminCommand({commitShardRemoval: "shard04"}). Commit rejects a shard that is not completely drained or still holds database-primary assignments or unsharded collections; successful commit returns ok. (medium confidence) (verified as of 2026-10-14) [source]
- Jumbo chunks can prevent draining from completing. Percona demonstrated a removeShard operation stuck with three remaining jumbo chunks in 2020; the current MongoDB removeShard reference likewise directs operators to clear jumbo flags once only jumbo chunks remain. The old blog is historical evidence, not an 8.3 cancellation runbook. (medium confidence) [source]
- Severalnines independently demonstrated shard removal as data migration to remaining shards and warned that those shards need enough free disk space. Its example uses the older removeShard API; use the current staged-command procedure for 8.3. (medium confidence) [source]
- Percona PMM documents a Draining Shards panel alongside Chunks Move Events for observing evacuation activity. Its wording refers to removeShard and does not establish validation of the 8.3 staged-command interface. (low confidence) (verified as of 2026-10-27) [source]
- MongoDB documents that draining prevents cluster backups and may close change-stream cursors that are not fully resumable. Cost-driven shard consolidation therefore needs an operational window that accounts for backup and consumer continuity. (medium confidence) (verified as of 2026-10-14) [source]
- Percona frames shard removal as a cost-reduction step after reducing data on high-cost shards; the staged MongoDB protocol still requires evacuation and a separate successful removal commit. Neither source supplies a general dollar saving or drain-time guarantee. (medium confidence) [source]
- db.adminCommand({startShardDraining: "shard04"}) requests asynchronous chunk evacuation by the balancer. The balancer must be enabled; with balancing disabled no chunks migrate and status continues to show remaining work. An ok response to start is not completion. (medium confidence) (verified as of 2026-10-14) [source]
- Poll db.adminCommand({shardDrainingStatus: "shard04"}) for progress and operator work: remaining chunk, database, jumbo-chunk and collection counts, plus dbsToMove and collectionsToMove. The documented terminal draining state is drainingComplete; it precedes explicit commitShardRemoval. (medium confidence) (verified as of 2026-10-14) [source]
- The documented shard-removal procedure does not require checkMetadataConsistency as an explicit commit gate. Its commit gate is completed draining and migrated collections/database primaries; catalog consistency checks are a separate maintenance diagnostic. (medium confidence) (verified as of 2026-10-14) [source]
- Percona PMM documents separate Draining Shards, Balancer Enabled and Chunks Move Events panels. It warns that sustained chunk migrations consume network and disk I/O and can increase replication lag; these panels provide operational context for the cost of evacuation. (low confidence) (verified as of 2026-10-26) [source]
- A nonzero draining-shard count should not be interpreted as proof of active migration: MongoDB says draining can remain pending while the balancer is disabled. Percona PMM exposes a separate Balancer Enabled panel, so inspect balancing state and migration activity alongside the draining count. (medium confidence) (verified as of 2026-10-14) [source]
Children
- Drain cancellation and in-flight migration races (frontier)
- Jumbo-chunk evacuation blockers (frontier)
- Receiving-shard capacity during scale-in (frontier)
- Shard-removal catalog consistency diagnostics (frontier)
Frontier under this node: Drain cancellation and in-flight migration races, Jumbo-chunk evacuation blockers, Receiving-shard capacity during scale-in, Shard-removal catalog consistency diagnostics