kafka-sink-connector
Parent: mongodb-kafka-connector · Published reference · snapshot 2026-09-08 · skill mongodb-operations-expert/references/mongodb-kafka-connector.md
↓ Facts as markdownall context files
The MongoDB Connector for Apache Kafka is a Kafka Connect plugin that bridges MongoDB and Kafka in both directions:
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Overview
- The MongoDB Connector for Apache Kafka is a Kafka Connect plugin that bridges MongoDB and Kafka in both directions: [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#overview]
- Source connector: MongoDB change streams → Kafka topics (CDC pipeline) [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#overview]
- Sink connector: Kafka topics → MongoDB collections (event consumer) [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#overview]
- Supports Confluent Platform, Confluent Cloud, Amazon MSK, and self-managed Kafka. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#overview]
Basic Change Stream Source
- This publishes change events to topic mongo.mydb.orders (format: <prefix>.<db>.<collection>). [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#basic-change-stream-source]
Resume Token Persistence
- The connector automatically persists the resume token in a Kafka Connect offsets topic. On restart, it resumes from the saved token. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#resume-token-persistence]
- If the token expires (oplog window exceeded during connector downtime): [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#resume-token-persistence]
Dead Letter Queue (DLQ)
- Configure DLQ to route failed messages instead of stopping the connector: [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#dead-letter-queue-dlq]
- DLQ messages include headers with error context. Process DLQ messages with a separate consumer for alerting or manual replay. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#dead-letter-queue-dlq]
Source Connector Errors
- ChangeStreamHistoryLost (error code 286): [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#source-connector-errors]
- The oplog has been truncated past the resume token. Resolution: [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#source-connector-errors]
- Set startup.mode=timestamp to start from a recent time [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#source-connector-errors]
- Or re-snapshot with startup.mode=copy_existing [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#source-connector-errors]
- Increase oplog size to prevent future occurrences [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#source-connector-errors]
- InvalidResumeToken: Resume token is corrupted or from an incompatible MongoDB version. Resolution: clear stored offset and restart connector. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#source-connector-errors]
Sink Connector Errors
- DuplicateKey (11000): Configure ReplaceOneDefaultStrategy instead of InsertOneDefaultStrategy to make sink idempotent. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#sink-connector-errors]
- DocumentValidationFailure (121): Kafka messages don't match MongoDB $jsonSchema validator. Check message schema vs collection validator. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#sink-connector-errors]
Sink Connector
- Worker parallelism: Set tasks.max equal to the number of Kafka partitions for the topic. [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#sink-connector]
CDC Pipeline Pattern: MongoDB → Kafka → Downstream
- For near-real-time with low latency: [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#cdc-pipeline-pattern-mongodb-kafka-downstream]
- Use poll.await.time.ms: 100 (shorter poll interval) [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#cdc-pipeline-pattern-mongodb-kafka-downstream]
- Monitor consumer lag on the Kafka topic [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#cdc-pipeline-pattern-mongodb-kafka-downstream]
- Keep poll.max.batch.size small (100-500) for lower latency at cost of throughput [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#cdc-pipeline-pattern-mongodb-kafka-downstream]
Anti-Patterns
- Single-partition topics with multiple sink tasks: Multiple sink tasks on a single partition = contention; match tasks.max to partition count [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#anti-patterns]
- Not configuring DLQ: Connector stops on first unprocessable message; always configure DLQ in production [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#anti-patterns]
- InsertOneDefaultStrategy for idempotent pipelines: Insert fails on duplicate; use ReplaceOneDefaultStrategy or BulkWriteStrategy for idempotent sinks [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#anti-patterns]
- Monitoring consumer lag but not resume token age: Consumer lag tells you about Kafka backlog; resume token age tells you about oplog risk (if token becomes invalid = full resync) [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#anti-patterns]
- Not increasing oplog for high-volume CDC: Connector outage exceeding the oplog window = full resync required; size oplog to cover expected maintenance windows [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#anti-patterns]
References
- MongoDB Kafka Connector Documentation [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#references]
- Source Connector Configuration [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#references]
- Sink Connector Configuration [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#references]
- Write Model Strategies [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#references]
- Kafka Connector GitHub [archived source: https://llms-explorer.com/sources/mdb-context-hub/mongodb-kafka-connector/#references]
Where this helps
- Consuming events from a Kafka topic and writing them into MongoDB collections as they arrive, instead of building a custom consumer loop by hand. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Building an idempotent write path when the same Kafka message might be redelivered, using ReplaceOneDefaultStrategy or BulkWriteStrategy instead of a plain insert that fails on duplicates. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Diagnosing why a sink connector stalls in production because a single malformed or schema-invalid message reached the topic, motivating a DLQ before that becomes an outage. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Sizing tasks.max for a sink connector against a topic's partition count when scaling ingestion throughput into MongoDB. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Project ideas
- Build a Kafka Connect sink pipeline that consumes an order-events topic and upserts into a MongoDB orders collection with ReplaceOneDefaultStrategy, then replay a message to confirm it doesn't create a duplicate. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Configure a DLQ for a sink connector and build a small consumer that alerts on new DLQ messages, using the error-context headers the connector attaches. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Provoke and observe a DocumentValidationFailure (121) by sinking a message that violates a MongoDB $jsonSchema validator, then fix the schema mismatch. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Tune tasks.max to match Kafka partition count on a test topic and measure the throughput difference against a mismatched, single-task configuration. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Common mistakes
- Running multiple sink tasks against a single-partition topic, creating contention instead of throughput — tasks.max should match the partition count. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Not configuring a DLQ, so the connector halts entirely on the first message it can't process instead of routing it aside for later inspection. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Using InsertOneDefaultStrategy for a pipeline that needs to be idempotent — inserts fail on any duplicate key instead of upserting cleanly. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Treating DuplicateKey (11000) errors as a data problem instead of a write-strategy problem — the fix is switching to ReplaceOneDefaultStrategy, not filtering the source data. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Known issues
- DocumentValidationFailure (121) means the Kafka message's shape doesn't match the target collection's $jsonSchema validator — this has to be reconciled by hand between the message schema and the MongoDB validator, the connector won't reconcile it for you. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- A single unprocessable message can stop the entire connector if no DLQ is configured, turning one bad message into a full pipeline outage. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- Worker parallelism is capped by topic partition count — setting tasks.max beyond the partition count doesn't add throughput, since only up to one task per partition does useful work. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
- The connector supports Confluent Platform, Confluent Cloud, Amazon MSK, and self-managed Kafka, but the sink write strategy (insert vs. replace vs. bulk) has to be chosen deliberately per use case rather than left at the default. [source] — AI-suggested, synthesized from this pack's existing facts — not extracted from a source document.
Children
- write-model-strategies (frontier)
- DLQ-error-handling (frontier)
- bulk-write-ordering (frontier)
Frontier under this node: DLQ-error-handling, bulk-write-ordering, write-model-strategies