Partitioner Strategies
Parent: MongoDB Spark Connector and Databricks Integration · Published reference · snapshot 2026-10-23 · skill mongodb-cohort-a04dd23fc577
↓ Facts as markdownall context files
Source-anchored research for Partitioner Strategies. Run status: COMPLETED.
These notes link each claim to its source. Some citations identify archived research reports that are no longer hosted on this site. A published reference means the content is available; it does not certify independent review or accuracy.Read the editorial policy and follow the sources before relying on a claim.
How it works
- Batch read partitioners create MongoDB range partitions that Spark can process in parallel. The documented batch partitioner configurations do not affect streaming reads. (high confidence) [source]
- Custom aggregation.pipeline stages must work with the selected partitioner; the current docs state that stages such as $group do not work when the partitioner creates multiple partitions. The current AutoBucketPartitioner implementation uses an initial $match while calculating bucket bounds, then appends the configured user pipeline to each generated partition pipeline. (medium confidence) [source]
Measurements and reference values
- AutoBucketPartitioner estimates documents per partition from average document size and its chunkSize target, samples documents, and uses $bucketAuto to derive key ranges. The documented defaults are a 64 MB chunkSize, 100 samples per partition, and _id as the partition field; average-sized targets do not guarantee equal-sized partitions. Compound field lists require MongoDB Server 7.0 or later. (high confidence) (verified as of 2026-10-14) [source]
- The current batch-read options also include SamplePartitioner, PaginateBySizePartitioner, PaginateIntoPartitionsPartitioner, ShardedPartitioner, and SinglePartitionPartitioner. The paginating configurations use a unique partition field; their documented defaults target 64 MB per partition or a maximum of 64 partitions, while SinglePartitionPartitioner creates one partition. (low confidence) (verified as of 2026-10-14) [source]
- A 2020 self-reported local benchmark using connector 2.4.1 and about 8.5 million documents totaling about 400 MB found the default/sample strategy fastest among the tested configurations and the single-partition strategy slowest because it removed parallelism. The author cautioned that this was a small test, so it does not establish a best strategy for current connector versions. (low confidence) [source]
Problems, failure modes and limitations
- The current documentation says ShardedPartitioner follows the shard configuration, but warns that it can be inefficient with MongoDB Server 6.0 or later and is incompatible with hashed shard keys. A 2019 Stack Overflow answer independently described the hashed-key limitation for the older MongoShardedPartitioner. (medium confidence) (verified as of 2026-10-14) [source]
Comparisons and alternatives
- A MongoDB Community forum thread from 2022–2023 reports that connector v10 partition statistics or partition-size calculation could take a long time; one respondent said a job remained in partition-size calculation after 40 minutes. The thread gives no query plan, dataset details, or resolution, so it is a historical symptom report rather than evidence of a current v11.1 defect. (low confidence) [source]
Facts and statements
- The current batch-read documentation names AutoBucketPartitioner as the default; the release notes say the default changed to AutoBucketPartitioner in connector version 10.5. (high confidence) (verified as of 2026-10-14) [source]
- Streaming reads use a separate change.stream.micro.batch.max.partition.count setting, documented with a default of 1 and limited to micro-batch streams; this is distinct from the batch-read partitioner option. (medium confidence) (verified as of 2026-10-14) [source]
- A Stack Overflow report of poor connector v10.1 read performance attributed the slowdown to Spark translating a downstream explode into an inefficient MongoDB aggregation; its author said no custom partitioner was in use and reported that checkpointing after the read avoided the issue. This case shows that connector slowdowns should not automatically be attributed to partitioner choice. (low confidence) [source]
Children
- No children recorded.