Streaming Write Semantics
Parent: MongoDB Spark Connector and Databricks Integration · Published reference · snapshot 2026-10-23 · skill mongodb-cohort-a04dd23fc577
↓ Facts as markdownall context files
Source-anchored research for Streaming Write Semantics. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
Definitions
- The Mongo connector's operationType is a separate control from Spark outputMode: outputMode selects which query rows are emitted, while operationType controls how an emitted row is written as a MongoDB document. The streaming write configuration lists replace as the operationType default, _id as the idFieldList default, and true as the upsertDocument default. (medium confidence) (verified as of 2026-10-14) [source]
How it works
- The MongoDB Spark Connector writes through Spark Structured Streaming's writeStream API with format mongodb and writer options for the connection URI, database, collection, checkpointLocation, outputMode, and trigger. MongoDB's guide says an omitted or non-Continuous trigger uses micro-batch processing and documents Trigger.Continuous(...) as its continuous-processing path. (high confidence) [source]
- Spark outputMode controls which result-table rows are emitted at each trigger: append emits rows that will not change in future triggers for stateful operators, update emits rows changed during the trigger, and complete emits the full result table. Query and sink compatibility can constrain the available modes. (high confidence) [source]
- The connector exposes checkpointLocation for streaming writes. Apache Spark describes checkpoints as persisted query progress and state used for recovery, and Databricks recommends a distinct persistent checkpoint directory for each query when restart recovery is needed. (high confidence) [source]
- A MongoDB Structured Streaming write uses writeStream.format("mongodb") with MongoDB connection, database, and collection options; outputMode selects the emitted results, trigger selects processing timing or mode, and start() launches the query. The MongoDB example also sets checkpointLocation. (high confidence) [source]
- Spark output modes describe which records operators emit per trigger: append emits rows that should not change in future triggers, update emits rows changed since the previous trigger, and complete emits the full result table. For stateful aggregations, append can wait for a watermark to make a result final. (high confidence) [source]
- For restart recovery, Spark stores query progress and state in a checkpoint location. MongoDB exposes checkpointLocation for streaming writes but lists its default as None, so configure an explicit durable path when recovery is required. (high confidence) [source]
- MongoDB's streaming-write guide maps Trigger.Continuous(...) to continuous processing and says other or omitted triggers use micro-batches. Apache Spark's current Structured Streaming page describes Real-Time Mode with Trigger.RealTime, requires update output mode, and limits supported sinks. These pages do not establish that MongoDB's documented continuous-write path is equivalent to Spark Real-Time Mode or that the MongoDB sink is supported there; treat compatibility as unresolved for the selected versions. (medium confidence) (verified as of 2026-10-23) [source]
Facts and statements
- Output modes are constrained by both the query and the sink. MongoDB's write guide demonstrates append and links to Spark's general outputMode API, but it does not state a Mongo-specific mode compatibility matrix; Databricks says sinks do not all support all output modes. Therefore, MongoDB sink support for update and complete is unconfirmed by these documents and should be checked for the deployed connector and runtime. (medium confidence) [source]
Children
- Spark outputMode versus MongoDB operationType (frontier)
- MongoDB Spark Connector compatibility with Spark Real-Time Mode (frontier)
Frontier under this node: MongoDB Spark Connector compatibility with Spark Real-Time Mode, Spark outputMode versus MongoDB operationType