Oplog sizing and embedded-verifier memory for migrations
Parent: Migration and Cutover Cost (Live Migration, mongosync, Dual-Running, Egress) · Published reference · snapshot 2026-10-31
↓ Facts as markdownall context files
Source-anchored research for Oplog sizing and embedded-verifier memory for migrations. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Definitions
- Oplog retention is a time-and-write-rate budget, not just a fraction of dataset size. Alibaba documents that frequent updates, insert/delete churn and batch updates can consume oplog history without comparable data growth; minimum retention can also make oplog storage exceed its configured size. (low confidence) [source]
How it works
- Alibaba documents that initial-sync retention must cover the data-copy duration and that heavy writes can grow the oplog beyond its configured size when minimum retention is enforced. Extending migration retention therefore also requires disk-capacity planning. (low confidence) [source]
Measurements and reference values
- Destination disk planning must include both logical migrated data and oplog entries created by the copy. MongoDB illustrates this with 10 GB of data plus another 10 GB for insert oplog entries; this is an example, not a universal storage multiplier. (high confidence) [source]
Problems, failure modes and limitations
- Change-stream migrations need source oplog entries to remain available until processed. MongoDB documents mongosync failure if unapplied changes roll off; Percona independently documents the same failure condition for PCSM. Large copies, pauses, and slow processing increase this risk. (high confidence) [source]
- Keeping the embedded verifier enabled requires sufficient destination oplog history. Shrinking that oplog can let verification fall behind and cause mongosync to error; destination-oplog reduction advice is explicitly conditional on disabling embedded verification. (high confidence) [source]
- MongoDB's embedded-verifier documentation budgets approximately 10 GB of RAM plus 500 MB per million migrated documents on the mongosync host. Insufficient available memory causes /start to return an error; the documented options are increasing host RAM or restarting with verification disabled. (medium confidence) (verified as of 2026-10-30) [source]
- The embedded verifier stores state in memory and cannot resume verification progress. Stopping or pausing sync and starting mongosync again restarts verification from the beginning, potentially leaving verification substantially behind migration progress. (medium confidence) (verified as of 2026-10-30) [source]
Comparisons and alternatives
- CloudSpinx reports a mongosync 1.22.0 lab process peaking at about 1.3 GB for roughly half a million documents, below MongoDB's sizing guidance. This single small, local test is an observation rather than evidence that production hosts can safely ignore the documented memory allowance. (low confidence) (verified as of 2026-10-26) [source]
- The standalone mongodb-labs migration-verifier persists change-stream resume tokens, unlike the embedded verifier's in-memory verification state. Its README still warns that excessive verifier lag can exceed oplog capacity and force verification to restart from the beginning. (medium confidence) (verified as of 2026-10-27) [source]
- Compare observed migration lag with the measured oplog window rather than relying on a fixed size alone. MongoDB and Percona both use db.getReplicationInfo().timeDiff and advise extending retention when migration lag approaches the window. (high confidence) [source]
Facts and statements
- For a sharded source, measure db.getReplicationInfo().timeDiff on every shard and use the smallest window. Compare it repeatedly with /progress lag.overallLagSeconds; increase retention or migration throughput before lag reaches that window. (low confidence) (verified as of 2026-10-14) [source]
- The embedded verifier's implementation designer describes per-collection maps of document identifiers and 64-bit hashes, plus buffered change events during the initial scan. These structures explain why verification needs much more memory than its final collection checksums. (low confidence) [source]
- Destination disk planning must include the migrated dataset and oplog entries generated while loading it. Both MongoDB and Percona document this additional destination storage requirement. (high confidence) [source]
Children
- No children recorded.