Batch Writes and Idempotency
Parent: MongoDB Spark Connector and Databricks Integration · Published reference · snapshot 2026-10-23 · skill mongodb-cohort-a04dd23fc577
↓ Facts as markdownall context files
Source-anchored research for Batch Writes and Idempotency. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Definitions
- For a batch sink, idempotency means that replaying the same input leaves the same sink state; it is separate from guaranteeing that each row executes only once. General Spark pipeline guidance emphasizes deterministic reruns and sink operations that are safe to apply again. The MongoDB Connector's keyed replace or upsert settings provide one way to make repeated identical documents converge. This last connection is an inference from the documented write semantics. (medium confidence) [source]
Structure and components
- MongoDB retryable writes handle eligible driver-level command retries by resending the supported command with the same transaction ID. A `bulkWrite` command is eligible only when its emitted write command contains no multi-document update or delete. Spark task retries can reissue sink writes outside that command retry identity, so driver retryable writes alone do not establish task-level idempotency. This distinction is an inference across the Spark and MongoDB retry contracts. (medium confidence) (verified as of 2026-10-23) [source]
How it works
- `ignoreDuplicatesOnInsert` defaults to `false` and is enabled only for unordered inserts with `operationType=insert` (`ordered=false`). In commit `c54b576`, the writer catches `MongoBulkWriteException` and suppresses it when the option is enabled and every entry in `getWriteErrors()` has code 11000; otherwise it rethrows. A test in that commit shows the setting suppressing duplicate-key errors on a unique `name` index for documents without input `_id`, although the current configuration docs say the data should include `_id` or the fields in `idFieldList`. The implementation's check does not inspect write-concern errors, so it does not establish their handling. The setting suppresses duplicate-key errors; it does not turn inserts into upserts. (high confidence) (verified as of 2026-10-23) [source]
- Spark task retries and MongoDB driver retryable writes are separate mechanisms. Spark's `spark.task.maxFailures` setting allows a failed task to be attempted again, while MongoDB retryable writes cover a documented set of acknowledged operations after network errors or primary elections, usually with one retry. Driver retryability alone therefore does not guarantee idempotency when a Spark task re-executes writes that already reached MongoDB. (medium confidence) (verified as of 2026-07-14) [source]
Comparisons and alternatives
- For repeatable batch writes, use a deterministic key in `_id` or `idFieldList` and `operationType=replace` or `update` with `upsertDocument=true`: the connector matches by the selected key and inserts when no match exists, so replaying the same key updates or replaces one document instead of adding another. The source rows must carry the same key on every replay. (medium confidence) (verified as of 2026-10-14) [source]
Facts and statements
- Spark can retry failed tasks. A Databricks Community report about MongoDB Spark Connector writes described duplicate documents after a task had partly written data and a retry repeated plain inserts without a stable unique target key. A clean or deduplicated upstream batch therefore does not by itself prevent duplicates at the sink. (medium confidence) [source]
- The current MongoDB Spark batch configuration defaults `operationType` to `replace`, `idFieldList` to `_id`, and `upsertDocument` to `true`. Replace and update operations match using the configured fields and insert if no match exists. As an inference from those semantics, replaying the same document with the same stable key converges on one matching document. (medium confidence) (verified as of 2026-10-23) [source]
- With the current documented default `truncateMode=DROP`, MongoDB Spark Connector's `overwrite` mode drops the target collection and creates a new collection with default options, removing a pre-existing custom unique index. If retry protection depends on that index, the index must exist during the load: creating it after the load leaves earlier writes without that constraint. `truncateMode=TRUNCATE` instead deletes collection entries while preserving indexes, collection options, and sharding configuration. A Databricks Community report describes creating indexes after overwrite and seeing duplicate documents after task retries. (medium confidence) (verified as of 2026-10-23) [source]
- With the current documented default `truncateMode=DROP`, MongoDB Spark Connector `overwrite` mode drops the target collection and recreates it with default options, removing a pre-existing unique index. If retries rely on that index, creating it after the load leaves writes during the load without the uniqueness constraint. The `truncateMode=TRUNCATE` option clears entries but preserves indexes, collection options, and sharding configuration. The accepted Databricks Community reply suggests writing to a staging collection and then atomically renaming it as a workaround; its reply does not establish that staging is duplicate-free or that rename removes duplicates. (medium confidence) (verified as of 2026-10-23) [source]
Children
- No children recorded.