Change-stream oplog-position and retry-error diagnostics
Parent: MongoDB Monitoring and Observability · Published reference · snapshot 2026-10-29
↓ Facts as markdownall context files
Source-anchored research for Change-stream oplog-position and retry-error diagnostics. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
How it works
- postBatchResumeToken represents scanned oplog progress even when no matching event is returned. The driver specification requires caching it for empty batches and when returning the batch’s last document. Using only the last matching event token can unnecessarily repeat scanning and leave the restart position much older than the server cursor position visible through optime. (medium confidence) [source]
- MongoDB 9.0 returns InterruptedDueToReplStateChange (11602) from getMore when an election makes the serving member incompatible with an exact primary or secondary read preference. Compliant drivers and mongos resume automatically, while manual getMore loops and mongosh must resume explicitly. The retriable interruption counter provides context for this behavior. (medium confidence) (verified as of 2026-10-14) [source]
Problems, failure modes and limitations
- Diagnose resume failure against retained oplog history: the oplog must retain the consumer's restart position. Repeatedly retrying a position already purged cannot reconstruct the missing history. Debezium documents error 286 (ChangeStreamHistoryLost) and connector recovery requiring a new snapshot or offset reinitialization; recovery is connector-specific. (high confidence) (verified as of 2026-10-27) [source]
- The driver specification distinguishes getMore failures from initial aggregate failures: an aggregate error is not eligible for automatic change-stream resumption. For server versions 4.4 and later, resumable server errors use ResumableChangeStreamError, with CursorNotFound specially allowed; network errors also qualify. A driver attempts one resume for a resumable failure, and can resume again after a subsequent successful resume and later failure. (low confidence) (verified as of 2026-10-27) [source]
- Resume with the same change-stream scope, pipeline and options that produced the token. MongoDB implementation documentation warns that changing a match filter can hide the resume event and make resumption fail; Debezium likewise requires matching scope, pipeline and options when using an emitted token to resume an independent stream. (high confidence) [source]
- SERVER-128753 reports an 8.0.18 getMore failure with code 280 and NonResumableChangeStreamError despite an allegedly retained, server-issued token. The reporter supplies oplog evidence dated 2026-06-05 and says treating the error as transient prolonged retries and connection buildup. This is a reported incident with an unconfirmed cause, not proof of a general server defect or a MongoDB 9.0 regression. (low confidence) (verified as of 2026-06-05) [source]
- Diagnostic inference: correlate per-cursor optime with error-counter changes and the application’s persisted resume offset. optime measures scanning, serverStatus classifies server-observed errors, and connector heartbeats expose a separate offset-commit concern. No one of these observations proves successful application processing or recoverability from its last durable token. (medium confidence) [source]
- In the new error counters, changeStreamHistoryLost counts unavailable oplog resume positions; changeStreamFatalError counts fatal non-resumable errors, including removing the event _id; bsonObjectTooLarge counts events exceeding the BSON size limit. Each category's other counter collects errors outside its named counters. (low confidence) (verified as of 2026-10-14) [source]
- The driver specification requires one resume attempt per resumable error and server selection using the original read preference. For server versions 4.4 and later, resumable server getMore errors use ResumableChangeStreamError, with CursorNotFound specially allowed; aggregate errors are not covered by this getMore resume rule. (low confidence) [source]
Comparisons and alternatives
- MongoDB 9.0 introduces metrics.changeStreams.error counters in serverStatus. nonRetriable contains changeStreamHistoryLost, changeStreamFatalError, bsonObjectTooLarge and other; retriable contains interruptedDueToReplStateChange and other. This taxonomy separates non-resumable failures from interruptions such as primary stepdown instead of treating every change-stream exception as the same retry condition. (high confidence) (verified as of 2026-10-27) [source]
- The nonRetriable.changeStreamFatalError counter includes fatal stream errors such as removing event _id in a pipeline; bsonObjectTooLarge counts events beyond the 16 MB BSON limit. The driver specification separately requires an error and stream closure when an event lacks its resume token. These failures require correcting the event or pipeline conditions rather than repeatedly replaying the same token. (high confidence) (verified as of 2026-10-27) [source]
Facts and statements
- MongoDB 9.0 adds cursor.changeStreams.optime to currentOp and the $currentOp stage. It identifies the most recent oplog entry scanned by a change-stream cursor, including entries that produce no output event, so it can be newer than the last event returned. Comparing this position with the newest oplog entry estimates server-side stream lag. (high confidence) (verified as of 2026-10-27) [source]
- The optime field updates once per getMore; scanning progress inside an unfinished getMore remains invisible. $currentOp reports idle cursors only with idleCursors:true, and a running getMore can initially lack its cursor document. These sampling and visibility rules must be checked before interpreting a missing or unchanged position as a stalled stream. (low confidence) (verified as of 2026-10-27) [source]
- Debezium documents a diagnostic trap for selective capture: the connector can keep reading changes in excluded collections while emitting no captured records, leaving committed Kafka offsets stale. heartbeat.interval.ms enables heartbeat messages that help commit progress in this situation; its documented default is 0, which disables them. A moving server cursor therefore does not by itself prove that a durable restart offset is current. (low confidence) (verified as of 2026-10-27) [source]
Children
- Per-cursor oplog scan lag sampling (frontier)
- ResumableChangeStreamError and terminal resume failures (frontier)
- Durable resume-offset freshness (frontier)
Frontier under this node: Durable resume-offset freshness, Per-cursor oplog scan lag sampling, ResumableChangeStreamError and terminal resume failures