Split change-stream event fragments and fragment-token recovery
Parent: MongoDB Replication · Published reference · snapshot 2026-10-27
↓ Facts as markdownall context files
Source-anchored research for Split change-stream event fragments and fragment-token recovery. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
How it works
- Fragment recovery retains normal change-stream constraints: keep the original pipeline and settings, treat tokens as opaque, and retain enough oplog history. MongoDB's implementation guide warns that changing $match can hide the resume event and cause ChangeStreamHistoryLost. (low confidence) [source]
How-to and procedures
- Use the split stage only in a change-stream pipeline, once, as its last stage. MongoDB recommends reducing event size with projection before resorting to splitting. (medium confidence) [source]
Measurements and reference values
- MongoDB 7.0 and the 6.0.9 backport introduce $changeStreamSplitLargeEvent for events exceeding 16 MB; it belongs once, last, in a change-stream pipeline. (medium confidence) [source]
- MongoDB introduced $changeStreamSplitLargeEvent in 7.0 and backported it to 6.0.9. It splits change events exceeding 16 MB; its syntax is {$changeStreamSplitLargeEvent: {}}. (high confidence) [source]
Problems, failure modes and limitations
- Splitting occurs between top-level fields, not recursively inside subdocuments. An indivisible field that exceeds the fragment limit can still cause an error; enabling the stage does not guarantee that every oversized event becomes consumable. (low confidence) [source]
- Debezium documents cursor.oversize.handling.mode=split: it assembles the fragments and uses the final fragment token for its offset and source.resume_token. Its default mode is fail. (low confidence) [source]
- Reassembling an event can produce an object exceeding MongoDB's BSON limit again. Driver design guidance warns that downstream BSON serialization or MongoDB storage may require splitting it anew. (low confidence) [source]
- SERVER-91406 records Location7182803 when a non-renameable $match blocks split-stage movement during resume processing. The ticket lists 6.0.15 as affected and fixes in 6.0.17, 7.0.13, 7.3.4, and 8.0.0-rc11. This is a historical version-specific defect, not evidence that all present split pipelines fail. (low confidence) (verified as of 2026-10-27) [source]
Comparisons and alternatives
- Drivers deliberately expose fragments rather than automatically joining them. Application code must handle partial sequences and failures during assembly. (medium confidence) [source]
Facts and statements
- Fragments arrive sequentially and carry splitEvent.fragment, a one-based index, and splitEvent.of, the total fragment count. MongoDB leaves reconstruction to the consumer; the C# guide illustrates merging fragment fields while excluding fragment _id and splitEvent fields. (high confidence) [source]
- Every MongoDB fragment has a resume token. Resuming after a non-final fragment returns the subsequent fragment; resuming after the final fragment returns the next event. A fragment checkpoint therefore records progress within an event, not necessarily completion of that event. (medium confidence) [source]
- Recovery design implication: if a consumer checkpoints a non-final fragment, it must durably preserve the preceding pieces to finish reconstruction after restart. An alternative is to checkpoint only completed events and replay the unfinished event from the prior completed-event token, with downstream replay handling. (medium confidence) [source]
- Resume with the original stream settings and pipeline. MongoDB's implementation guide warns that changed settings can make the original event token impossible to find. (medium confidence) [source]
- Debezium also documents matching full-document options during startup token validation and snapshot-position selection; a split first event is fully consumed before its final token becomes the snapshot start position. (low confidence) [source]
- A fragment need not carry operationType: the Java driver's getter explicitly permits null for split events. Consumers should not assume every fragment has complete-event metadata. (medium confidence) [source]
- SERVER-85707 records mongos assertions from the split stage setting the internal $_generateV2ResumeTokens flag; affected versions are 6.0.12 and 6.0.13, with a fix in 6.0.14. (low confidence) (verified as of 2024-02-19) [source]
- SERVER-91406 records Location7182803 with a non-renameable $match and fragment resumption. Listed fixes include 6.0.17, 7.0.13, 7.3.4 and 8.0.0-rc11; adding the split stage alone did not avoid this optimizer defect. (low confidence) [source]
- Debezium's stable documentation specifies cursor.oversize.handling.mode=split for reconstruction into one event, using the last fragment's resume token as the offset. It also documents matching streaming full-document and pre-image options during startup token validation, and consuming every fragment before recording a split snapshot-start event's final token. (low confidence) (verified as of 2026-10-27) [source]
Children
- Partial-fragment durable assembly and checkpoint atomicity (frontier)
- Pipeline-stable fragment resumption (frontier)
- Oversized-event field boundary failures (frontier)
- Durable partial-fragment assembly and event checkpoint atomicity (frontier)
- Fragment sequence validation across consumer restarts (frontier)
Frontier under this node: Durable partial-fragment assembly and event checkpoint atomicity, Fragment sequence validation across consumer restarts, Oversized-event field boundary failures, Partial-fragment durable assembly and checkpoint atomicity, Pipeline-stable fragment resumption