Sync pipeline cost: CDC (Debezium, Kafka, Monstache) vs native change streams
Parent: Platform Consolidation TCO (Atlas Search/Vector Search vs Separate Search or Vector DB) · Published reference · snapshot 2026-10-31
↓ Facts as markdownall context files
Source-anchored research for Sync pipeline cost: CDC (Debezium, Kafka, Monstache) vs native change streams. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Structure and components
- In Kafka Connect deployment, Debezium publishes events to Kafka topics that multiple clients can consume independently, and distributed Kafka Connect provides connector fault tolerance. A direct native consumer must be compared against those required capabilities when estimating infrastructure and maintenance cost; removing components alone does not establish equivalent service. (medium confidence) [source]
How it works
- A change stream opened through mongos creates streams on every shard, even when it targets a particular shard-key range; mongos sorts and filters results. Inference: a shard key that targets ordinary application queries does not automatically eliminate cluster-wide CDC fan-out. (medium confidence) [source]
- Full-document lookup is a distinct work and correctness choice: updateLookup reads the document from the collection, and rapid subsequent updates can make the returned document newer than the triggering event. Debezium likewise warns that updateDescription and after can disagree. Prefer delta capture when the downstream consumer can correctly apply deltas; benchmark the extra work when full documents are required. (high confidence) [source]
- Monstache direct-read concurrency can stress MongoDB; increasing collection splitting increases connections and process memory. Its stateful direct-read option records completed collections and skips them on subsequent restarts, avoiding repeated full copies unless that state is cleared. (low confidence) [source]
- Resume requires retained oplog history. Debezium documents ChangeStreamHistoryLost when the saved position expires and describes recovery paths including a new snapshot. Inference: budget outage tolerance, retention and possible recopy work for both connector and native designs. (high confidence) [source]
Problems, failure modes and limitations
- Debezium documents loss of resumability if the required oplog position expires: recovery depends on snapshot.mode, and initial mode can fail when an old offset remains. Cost planning should include oplog retention and potential re-snapshot reads, network transfer, and downstream writes; durable Kafka history does not restore events that never reached Kafka. (medium confidence) [source]
- MongoDB warns that change streams cannot use indexes and that many specifically targeted streams can affect server performance. Each stream also holds a getMore connection; exceeding the connection-pool capacity can increase notification latency. A native-consumer design should therefore measure watcher count and pool capacity as cost and latency inputs. (high confidence) [source]
- Monstache issue 374 records a 2020 report on Monstache 5.0.11 and MongoDB 4.0 of tailing stopping, BSONObjectTooLarge in mongos logs, and later 'Resume point not in the oplog'. The thread discusses batch sizing and subsequent clustered-worker problems. Treat it as historical failure evidence for recovery tests, not proof that a current Monstache version has the same defect. (low confidence) (verified as of 2020-05-22) [source]
Comparisons and alternatives
- Kafka is optional for Debezium: its MongoDB connector can be embedded without Kafka or Kafka Connect, with progress persistence owned by the application. Kafka infrastructure cost must therefore be attributed to the chosen deployment, rather than assumed inherent to Debezium. (high confidence) [source]
- Estuary's June 4, 2026 engineering write-up reports that batch prefetching and faster BSON-to-JSON processing increased its connector throughput from 34 to 57 MB/s with approximately 20 KB documents, and from approximately 6 to 17.5 MB/s with 250-byte documents. These are vendor-reported before/after results for its connector, not Debezium-versus-native or dollar-cost benchmarks. (low confidence) (verified as of 2026-06-04) [source]
- A full-document lookup need not represent the document at the triggering update: later updates can intervene. A deletion or shard-key change can also leave the lookup without a document. Consumers must accommodate these semantics when choosing cheaper delta events versus full-document synchronization. (high confidence) [source]
Facts and statements
- Debezium's MongoDB connector delegates oplog capture and decoding to MongoDB change streams; Monstache's version-6 README also describes change streams as its default. Comparing these tools with a native consumer therefore compares additional pipeline machinery around a shared capture API. (high confidence) [source]
- Debezium's March 8, 2023 release post explains that moving MongoDB include/exclude filtering to the server avoids sending discarded events across the network and reduces connector CPU work. This is historical evidence for measuring filter placement, not a guarantee of a particular saving on a current deployment. (low confidence) [source]
- A Kafka Connect deployment adds a separate Connect service beside Kafka brokers and can route events through sink connectors. Debezium Engine instead embeds connectors in a Java application without requiring Kafka or Kafka Connect clusters; broker deployment is therefore optional for Debezium. (low confidence) [source]
- Full-document capture can require an additional document lookup. Debezium supports update events without full documents, while MongoDB advises limiting lookup work. Inference: compare delta events with full-document events before attributing source CPU or network cost solely to the connector. (high confidence) [source]
Children
- CDC recovery and oplog retention budgeting (frontier)
- Full-document lookup versus delta capture (frontier)
- Change-stream fan-out and watcher consolidation (frontier)
- Durable replay versus direct-consumer operational ownership (frontier)
Frontier under this node: CDC recovery and oplog retention budgeting, Change-stream fan-out and watcher consolidation, Durable replay versus direct-consumer operational ownership, Full-document lookup versus delta capture