Atlas Stream Processing resumable initial sync and source retention
Parent: MongoDB Atlas Stream Processing · Published reference · snapshot 2026-10-29
↓ Facts as markdownall context files
Source-anchored research for Atlas Stream Processing resumable initial sync and source retention. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
How-to and procedures
- Configure an oplog window of at least 24 hours for an ASP change-stream source and longer than the longest expected interruption. The operations guide warns that downtime beyond retention can cause ChangeStreamHistoryLost; 24 hours is not a guarantee for longer outages. (high confidence) (verified as of 2026-10-14) [source]
Problems, failure modes and limitations
- ASP checkpoints initial sync for every supported _id type. Unpredictably ordered identifiers may slow copying without removing checkpoint recovery support. (low confidence) (verified as of 2026-10-14) [source]
- A checkpoint cannot restore oplog history that has rolled off the source. An outage beyond the retained history can produce ChangeStreamHistoryLost; size the oplog window for acceptable processor downtime. (high confidence) (verified as of 2026-10-14) [source]
- If the required resume token has left the oplog by initial-sync completion, ASP fails to avoid data loss. Restarting with resumeFromCheckpoint=false reruns initialSync; checkpoint recovery cannot reconstruct expired oplog history. (high confidence) (verified as of 2026-10-14) [source]
- Independent connector evidence supports the retention hazard: Estuary says expired oplog events can require a fresh backfill, while Confluent documents resume failure after the saved token disappears. These describe those connectors, not ASP initial-copy internals. (high confidence) (verified as of 2026-10-27) [source]
- Confluent's MongoDB Atlas Source Connector guidance also identifies sparse namespace updates as a stale-token risk and recommends connector heartbeat settings. These settings are Confluent connector controls, not documented ASP configuration. (low confidence) (verified as of 2025-01-13) [source]
Comparisons and alternatives
- Checkpoints store source positions rather than copies of source documents. Deleting or expiring source documents after a committed checkpoint does not itself block resumption; retention of the relevant oplog history remains necessary. (low confidence) (verified as of 2026-10-14) [source]
Facts and statements
- With initialSync enabled, ASP checkpoints both collection copying and subsequent change-event processing. It records an initial oplog position, partitions the copy, and replays changes after copying; interruption can resume from a checkpoint. (low confidence) (verified as of 2026-10-14) [source]
- During multi-collection initial sync, a restarted processor skips completed collections and resumes an unfinished collection from its last synchronized document; previously emitted documents can be redelivered. (low confidence) (verified as of 2026-10-14) [source]
- Initial-sync recovery is at-least-once: collection-copy or catch-up processing can duplicate documents. Sinks should handle repeated documents idempotently. (high confidence) (verified as of 2026-10-14) [source]
- Default ObjectId or ordered int/long identifiers provide optimal initial-sync performance. Other supported identifiers can extend the copy phase, and a network interruption on a growing partition can prolong it further, increasing exposure to oplog expiry. (low confidence) (verified as of 2026-10-14) [source]
- The source replica-set member initial sync is distinct from ASP initialSync. A member performing replica-set initial sync truncates its own oplog, which can affect dependent change streams and ASP checkpoints. (low confidence) (verified as of 2026-10-14) [source]
- A public ASP-to-Iceberg demo reports duplicate Athena rows after a restart without a checkpoint reruns initialSync. This is a demo-specific troubleshooting report and does not establish that all ASP sinks use insert-only replay. (low confidence) (verified as of 2026-10-27) [source]
- initialSync emits preexisting collection documents as insert change events before ingesting new change events; a successfully completed initial sync does not normally repeat. (low confidence) (verified as of 2026-10-14) [source]
Children
- No children recorded.