Spark Connector V10 Architecture

Parent: MongoDB Spark Connector and Databricks Integration · Published reference · snapshot 2026-10-23 · skill mongodb-cohort-a8bd98441f1a

↓ Facts as markdown all context files

Source-anchored research for Spark Connector V10 Architecture. Run status: COMPLETED.

These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.

Problems, failure modes and limitations

MongoScanBuilder in 10.5.0 implements filter pushdown and required-column pruning. It translates supported filters to an aggregation $match and returns filters it cannot translate for Spark to process. (high confidence) [source]
A MongoDB community response dated February 2, 2024 states that v10.x does not expose direct RDD support in answer to a 10.1.1 document-RDD question. This concerns the connector API and must not be expanded into a claim that Spark itself cannot derive an RDD from a DataFrame. (low confidence) [source]

Comparisons and alternatives

Connector 10.x is a rewrite around Spark Data Source V2. Its provider namespace is com.mongodb.spark.sql.connector.MongoTableProvider, rather than the legacy com.mongodb.spark.DefaultSource; the distinct namespaces allow both connector generations in one application, but do not make 10.x a drop-in API replacement. (high confidence) [source]
In connector 10.5.0, both streaming scan implementations use MongoDB change streams; streaming therefore represents database change events rather than repeated full-collection batch reads. (high confidence) [source]
The v10.x streaming configuration documents an explicit schema when change.stream.publish.full.document.only is false. When true, the connector emits fullDocument values, filters out events without fullDocument, and can infer a schema if none is supplied; this changes the output record shape rather than merely adding an update lookup. (low confidence) (verified as of 2026-10-23) [source]
MongoDB changed connector versioning at V10: the connector major version no longer mirrors the supported Spark version. Compatibility must come from the connector's compatibility table rather than matching major numbers. (high confidence) [source]

Facts and statements

The V10 API models a MongoDB collection as MongoTable. MongoTableProvider supplies the table, which creates MongoScanBuilder and MongoWriteBuilder instances for reads and writes. (high confidence) [source]
In r10.5.0, MongoScan holds the read schema and configuration, then produces MongoBatch, MongoMicroBatchStream or MongoContinuousStream according to the execution mode. This follows Spark 3.5.6 Scan, which separates a logical scan from its physical batch or streaming representation. (high confidence) [source]
In connector 10.5.0, MongoScanBuilder appends its translated filter pipeline after the user-configured aggregation pipeline and constructs the scan with the pruned schema. (low confidence) [source]
The 10.0.5 batch API plans MongoInputPartition objects that split the collection and uses a MongoPartitionReaderFactory to create row-based readers. BSON-to-row conversion occurs through BsonDocumentToRowConverter. (low confidence) [source]
The 10.0.5 write API separates MongoBatchWrite from MongoStreamingWrite. Its streaming writer factory is serialized and sent to executors. (low confidence) [source]
Batch partitioners must preserve custom aggregation semantics: a global stage such as $group is incompatible with a partitioner that creates multiple partitions. Batch partitioner settings are separate from change-stream micro-batch partitioning. (high confidence) [source]
A firsthand 2023 report reproduced DataType.sameType NoSuchMethodError inside InferSchema with connector 10.1.0–10.2.0 on Databricks 13.0–13.2, despite successful local Spark 3.4.1 reads. The reporter stated on September 19, 2023 that Runtime 13.3 LTS fixed it; this is a historical binary-compatibility incident, not evidence that V10 reads remain broken. (low confidence) [source]
A firsthand V10.1 issue report attributed a workload regression to a generated MongoDB query after a downstream explode and reported that checkpointing the Dataset resolved it. This is an anecdotal attribution, not proof that V10 universally pushes explode into MongoDB. (low confidence) [source]
In release r10.5.0, MongoTable implements Spark Table, SupportsRead and SupportsWrite. It constructs MongoScanBuilder for reads and MongoWriteBuilder for writes, and declares batch read/write, micro-batch read, continuous read, streaming write, truncate and ACCEPT_ANY_SCHEMA capabilities. (high confidence) [source]
In r10.5.0, MongoBatch delegates partition planning to generateMongoBatchPartitions and creates MongoBatchPartitionReaderFactory with a BsonDocumentToRowConverter. The batch read boundary therefore converts MongoDB BSON documents into Spark rows under a StructType schema. (low confidence) [source]
In r10.5.0, MongoScanBuilder translates convertible Spark filters into a MongoDB aggregation $match stage and returns unsupported filters for Spark to evaluate. It appends the generated stages after the configured aggregation pipeline and passes a pruned schema into MongoScan; Data Source V2 does not imply that every expression is pushed down. (high confidence) [source]
In r10.5.0, the write path uses MongoWriteBuilder.buildForBatch() to create MongoBatchWrite and buildForStreaming() to create MongoStreamingWrite. MongoTable exposes both write capabilities through the same table abstraction. (high confidence) [source]
The release history scopes the streaming architecture: 10.0.0 introduced Structured Streaming support, while 10.1.0 added micro-batch support. The r10.5.0 implementation explicitly supplies both micro-batch and continuous read implementations, so capabilities should be attributed to a specific 10.x release. (high confidence) [source]
In r10.5.0, streaming reads use MongoDB change streams. Multiple-collection streaming is supported, but MongoScan.toBatch() rejects configurations selecting multiple collections or all collections; streaming collection selection must not be assumed to work for batch reads. (high confidence) [source]
For full-document schema inference across multiple streamed collections, the v10.x documentation says inference happens at stream startup and excludes collections created later. MongoScan also carries a StructType into its streaming reader implementations; a schemaless source does not establish automatic schema evolution in the Spark stream. (high confidence) (verified as of 2026-10-23) [source]
Databricks independently illustrates the MongoDB connector through Spark DataFrame reader and writer calls using format("mongodb"), connection.uri, database and collection, which corroborates its Spark data source interface. (low confidence) [source]

Children

Data Source V2 scan lifecycle (frontier)
BSON and Spark SQL schema conversion (frontier)
Change-stream offsets and checkpoint recovery (frontier)
Data Source V2 table and scan lifecycle (frontier)
BSON to Spark row schema conversion (frontier)
Change-stream event and full-document schemas (frontier)

Frontier under this node: BSON and Spark SQL schema conversion, BSON to Spark row schema conversion, Change-stream event and full-document schemas, Change-stream offsets and checkpoint recovery, Data Source V2 scan lifecycle, Data Source V2 table and scan lifecycle

← the whole tree · 3D view· how to read this page