CDC-patterns

Parent: mongodb-kafka-connector · researched 2026-05-28T16:41:46.364Z· 4 sources · 4 concepts · skill mongodb-kafka-connector

The MongoDB Connector for Apache Kafka is a Kafka Connect plugin that bridges MongoDB and Kafka in both directions:

Overview

The MongoDB Connector for Apache Kafka is a Kafka Connect plugin that bridges MongoDB and Kafka in both directions: [source]
Source connector: MongoDB change streams → Kafka topics (CDC pipeline) [source]
Sink connector: Kafka topics → MongoDB collections (event consumer) [source]
Supports Confluent Platform, Confluent Cloud, Amazon MSK, and self-managed Kafka. [source]

This publishes change events to topic mongo.mydb.orders (format: <prefix>.<db>.<collection>). [source]

The connector automatically persists the resume token in a Kafka Connect offsets topic. On restart, it resumes from the saved token. [source]
If the token expires (oplog window exceeded during connector downtime): [source]

Configure DLQ to route failed messages instead of stopping the connector: [source]
DLQ messages include headers with error context. Process DLQ messages with a separate consumer for alerting or manual replay. [source]

ChangeStreamHistoryLost (error code 286): [source]
The oplog has been truncated past the resume token. Resolution: [source]
Set startup.mode=timestamp to start from a recent time [source]
Or re-snapshot with startup.mode=copy_existing [source]
Increase oplog size to prevent future occurrences [source]
InvalidResumeToken: [source]
Resume token is corrupted or from an incompatible MongoDB version. Resolution: clear stored offset and restart connector. [source]

DuplicateKey (11000): [source]
Configure ReplaceOneDefaultStrategy instead of InsertOneDefaultStrategy to make sink idempotent. [source]
DocumentValidationFailure (121): [source]
Kafka messages don't match MongoDB $jsonSchema validator. Check message schema vs collection validator. [source]

Worker parallelism: Set tasks.max equal to the number of Kafka partitions for the topic. [source]

For near-real-time with low latency: [source]
Use poll.await.time.ms: 100 (shorter poll interval) [source]
Monitor consumer lag on the Kafka topic [source]
Keep poll.max.batch.size small (100-500) for lower latency at cost of throughput [source]

Single-partition topics with multiple sink tasks: Multiple sink tasks on a single partition = contention; match tasks.max to partition count [source]
Not configuring DLQ: Connector stops on first unprocessable message; always configure DLQ in production [source]
InsertOneDefaultStrategy for idempotent pipelines: Insert fails on duplicate; use ReplaceOneDefaultStrategy or BulkWriteStrategy for idempotent sinks [source]
Monitoring consumer lag but not resume token age: Consumer lag tells you about Kafka backlog; resume token age tells you about oplog risk (if token becomes invalid = full resync) [source]
Not increasing oplog for high-volume CDC: Connector outage exceeding the oplog window = full resync required; size oplog to cover expected maintenance windows [source]

Frontier under this node: debezium-handlers, exactly-once-semantics, outbox-pattern